I live in Victoria, cruise ships do NOT dock in the inner harbour, they dock at Ogden Point, which is on the Strait of Juan de Fuca. USA is right across the strait, you can see lights from Port Angeles, in the evening. However, there are 2 ferries (one to Port Angeles and one to Seattle) that DO dock in the Inner Harbour, near the Parliament Buildings.
Of course, you can, I live in Victoria and I have done it. The correct procedure is to contact HAL and ask for a 'deviation request'. If approved, you will be given a disembarkation time AFTER all other guests have departed for their tours. I was on a 14-day round-trip Alaskan cruise, which originated in Seattle. Victoria, which meets the requirements for the Jones Act, was the last port prior to returning to Seattle. It simply made no sense for me to stay onboard and then fly home the next day from Seattle. The whole process worked like a charm 🙂