Drop off/Pick Up NYC by private car/walking difficulty

We are being driven to the port in Manhattan by a private driver. One of the passengers has difficulty walking and uses a walker. Can we get dropped off right at the terminal? Also, where do we get picked up? I remember having to cross the street before embarking and after disembarking in order to get a taxi last time. The taxi driver said no vehicles were allowed on the pier (yet we could see other vehicles going up the ramp).

You can definitely be dropped off right at the pier. Your taxi driver was wrong if he/she told you otherwise.

It's very common for all disembarking passengers to be directed to the other side of 12th Ave. and drivers from car services as well as private vehicles are often prevented from entering the terminal for pick ups.