Salgal, are you planning to do this while in Vancouver port just for the day, or are you planning this as pre or post cruise trip. I live in Victoria and you need to know the logistics involved. The trip form the port in Vancouver to the ferry terminal to the island is about an hour, can be more during rush hour, the ferry is 1 hr 35 minutes and then the drive into Victoria is about 40 minutes,( only 30 minutes to the Butchard Gardens) .
Victoria is beautiful, and totally worth the visit,the gardens are incredible( and I'm not even a " garden " person) .
Yes Sam's deli is famous, the best sandwiches in the country for sure, ( and HUGE) .
Also Murchies for cakes and coffees and teas.
Rogers Chocolates( hand made in store) .
The Inner harbour of the city is lovely , you don't need a car once you are there , the Empress hotel, the Royal British Columbia Museum( yes worth seeing, great sections on our First Nation people) the Parliment buildings and shopping are all with in walking distances.
It is all doable in a day, but a long day, I would recommend taking the ships tour if you are only in Vancouver port for the day. If doing it on your own , then I think taking a bus tour to Victoria and Butchard Gardens is a great idea. Greyline bus company probaly has tours that you can take that are cheaper then the crusie line sponsored pre or post cruise tours.
Tthe other option is renting a car, ( or in you case two mini vans. ) the driving is very sane here, so not scary at all.