Let's try a bit of math...
Maximum capacity is 166 guests, while usually (especially this year) mostly are 130 - 140 guests are on board. After about 15 cruises with Star Clippers, in my experience about 20 % are travelling in organised groups or with family members or friends. 110 guests are remaining.
Overall, about 30 % are native english speaking guests, interested on english narrated tours. Maybe a bit more on your cruise, and maybe some of other mother tongues could be interested. So, a maximum of 50 people are left.
Back-2-back-cruisers are not much (last cruise we had 2 on board, the only time we went B2B we were 3 in total). Then there are only 4 - 6 people left. That are 2 - 3 couples, and you are already one of them. And this is very optimistic counting. It is more likely that you will be the only native english speaking couple doing these two cruises B2B.
The stop in Athens inbetween is ideal for sightseeing in Athens to avoid an additional night at pre- or post-cruise-stays. I will bet none of all the B2B-cruisers is interested to spend this day out of Athens.
And guests who will board the ship at 4 p. m., if they are already in Athens and not flying in the same day, will surely not bring their luggage early in the morning to the port, just to leave Athens for that day. They rather want to have a relaxed breakfast at their hotel, strolling a bit around in Plaka or in the Acropolis area, having a light greek lunch, packing their suitcases and going to reach the boarding of the ship just in time. No chance to attract them. (By the way. We did so in June, and during our stroll through Plaka, we randomly met the only four B2B-cruisers from the cruise before and our cruise in a street café - we recognised them by their shirts and bags. Yes, the sloop shop has nice shirts... 😉 )
People whose cruise ends and leave the ship at that day are heading to the airport or to their booked hotel, at least to store their luggage there to have their hands free for sightseeing. (That may even be the reasons for you not to organise this specific excursion on the day of embarcation or debarcation.)
My bet: You will have a wonderful excursion on your own. Enjoy it. You are doing the same like most other guests: You will have a wonderful individual vacation with a great sailing adventure, together with some nice other people, who even look to realise their independent plans around the cruise.