I did this cruise on the same ship this past August. Yes, you can embark either day. If you fly in on the 4th, or before (which I highly recommend), then I recommend going ahead and board on the 4th. Use the cabin as your hotel that night.
Keep in mind, if you have a connection on Toronto, there are only so many flights to Quebec each day. We were supposed to arrive in Quebec at 5:30pm on Wednesday. We didn't get there until 9:30am Thursday morning. Also, keep every single piece of your flight paperwork. If there a delay, and you're flying Air Canada, you may be eligible for reimbursement. My partner and I both boot l got 1k CAD, although mine wasn't without a fight.