First of all I think that the Gardens are a great place to visit if you are in the area. But the whole area is beautiful and the inner harbour are is just a start. Normally all cruise lines have a bus laid on that will take you downtown/inner harbour for a small fee. You can walk in about 20 mins though if you wanted. All flat through residential area.
You ship's time in port sometimes has a lot to do with where you are coming from and/or going to because of US law in relation to starting and ending ports of your specific cruise. YOu should be able to figure out the proposed hours of your stay by checking your itinerary on line or with your TA.
If you are arriving in the evening I would suggest for sure that you arrange to do the tour via the Ship. An evening stop usual means limited time and sometimes a hurried/late arrival from the North although this is not always the case, A ships tour will get you there and back to the beat of your ships schedule.
If you have a longer stay you can easily get tours from the All attraction Ticket booth in the Tourist Centre across from the Empress hotel in the Inner harbour. Usually not far from where your bus will drop you.
Hope that helps to start. Let us know if you have any other details you might want clarified.
The gardens are actually about 23 Kms.(12.5 miles) North East of the City of Victoria via good roads. Too far for a taxi in Canada to be ressonable.