We did end up booking with Cay Tours for this. There was confusion regarding our departure time and then a delay leaving the port was added on top of it. We might have been on the same bus as you if you were there the same day. I know they brought us all to the caves first and then they brought others that weren't doing the cave to their other destinations after. The majority of the cave people took the bus directly back to port but we four had St. Pedro added to the trip so a separate driver was there to bring us to the castle. We were worried about getting back to port in time since we were running late due to the morning issue but he made sure to give us a brief and personal shortened tour of the castle so we didn't miss our departure.
While it started out rocky, he made up for it in the end. No, we didn't get as much time at the castle but he did what we asked which was to make up for lost time and put our minds at ease about not missing the boat.
I'm sorry about your deal. I would be mad too. I noticed the bus driver passed the other two destinations and went straight to the caves but at the time I didn't realize that anyone on the bus wasn't doing the caves. Perhaps had they been on time, they would've dropped you off first. Not sure. I know that the caves have set tour times so perhaps because we were running late he decided to go straight there first.
In the end, it worked out for us even with the shortened castle tour but the company does lack organization and I'd probably use someone else next time too.