Sorry but the service in the Haven restaurant far exceeds service at any specialty dining venue (in my experience). I’d be upset. For the OP, if you want the sanctuary of the Haven (and I don’t blame you) have your butler bring your said specialty restaurant dinner to the Haven or your cabin. Especially if you can’t get a dining time you like.