Some ports I would feel extremely safe doing a self guided tour or finding an excursion in the port (Aruba, Grandy Cayman, Cozumel, San Juan, St Thomas), but others there is no way I do anything like that (Jamaica, Belize, Honduras, DR (any of the ports), Nassau. They may be perfectly safe, but infrastructure alone can cause issues trying to get back to the port if something happens. In those situations is it best to have the guarantee from the cruise line.
Anything can happen anywhere, but to quote Jurassic Park "If the pirates of the Caribbean breaks down, the pirates don't eat the tourists!" Just be sure to do your research and know your time limitations.