As others have said Bayonne/Cape Liberty is almost exclusively Royal territory.
However, if your ship is really docking there then Newark is the better choice and staying near the port makes sense. There is a lot of public transportation that goes into Manhattan from that area in NJ if you want to go in for a couple of hours. The area's public transportation can be daunting for the uninitiated.
However, assuming that the ship leaves from Manhattan (the most likely scenario) then even if it is a few dollars more you are better off staying in Manhattan especially if the embarkation day is a business day (ie, normal weekday). Manhattan is an Island (yes, I am stating the obvious), and on business days the commuter traffic can be grueling (limited routes into Manhattan), and you may not want to stress out on your departure day.