SF: Remove layer map from Client

Cleanup before layer refbase removal. We
can lookup the layer from the handle
instead of maintaining a map.

Bug: 238781169
Test: presubmit
Change-Id: Ie6543e4c344359be6a221c164d193656e9190cc8
4 files changed