We just returned from our cruise and I am happy to report that we walked from the cruise ship terminal to Castillo San Felipe de Barajas in Cartegena - and back! The ship arrived at 12 noon and we disembarked fairly quickly. It was a hot day - something like 91C with high humidity, so it was like walking in an oven. We were offered a taxi many times but we declined and set off on foot.
The traffic was moderately busy but you have to be on your guard at all times. The sidewalk is unpredictable - narrow at times and lots of stepping up or down. Also, there are parts where the sidewalk is in poor shape, so watch out for holes or pipes sticking out. We were the only ones that we could see walking, however we felt quite safe the whole way (we were approached a couple of times by people trying to sell a few candies for a dollar but they were not very persistent).
As we had been to this port before, we really only wanted to spend our time at the fortress, however, if we had more time we could have walked another 15 minutes or so to the Old Town. Taxis are plentiful and easy to hail so we felt like we could easily get a ride if we needed one. It really helped that we had scouted out the route on google maps before we left (and printed detailed copies to take with us) as it was easy to get turned around when you first leave the terminal.
Would we recommend walking - yes, and no. It is definitely doable - it only took about 35 minutes each way and is not a long walk. However, just make sure you are prepared for the heat and conditions of the sidewalk and keep your wits about you - and be prepared to sweat off some of those trips to the buffet on the ship!