Well Victoria Island is in the High Arctic, so you're going to need more than one day to get there.
Vancouver Island, where the city of Victoria is located, is easy to get to independently. You could rent a car and take the ferry over and back - but that is going to be a 14 hour day and ferry reservations must be made.
There is the BC Ferry Connector bus which drives right onto the ferry and takes you into downtown Victoria. Then you'd have to taxi out to the Gardens, so another long day.
https://bcfconnector.com/
With only one day, the easiest would be fly Harbour Air there and back, expensive but an easier day with great views.
https://harbourair.com/tours/spend-the-day-in-victoria/
Or just spend a relaxing day in Vancouver visiting all the lovely gardens there.