Santa Barbara Shellfish Company on Stern's Wharf is where you want to go. Never had any there myself, but once saw some people eating some outside the restaurant. Take Lil Toot, the water taxi from the harbor and it can bring you across to the wharf if you didn't want to walk from the harbor to the wharf.
The SB Fish Market might have some. It's in the harbor, past Brophy's and down one of the side alley ways, but I don't think they do a sit down menu or offer anything more than just by itself.