Interestingly, the only time we've been required to take our passports into a Caribbean port was in San Juan - a U.S. territory. No explanation was given but pax were told that if they debarked in SJ to be sure and bring your passports because they would be checked before allowing you to re-board. Sure enough, security were checking both Set Sail cards and passports. Perhaps a one time situation, I don't know but at least we knew ahead of time.