Within walking distance of the port the Standard Baking Co has wonderful breads, croissants, cookies & brownies. Also close by the ferry over to Peaks for a walk around or golf cart rental.
If you are not into any physical activity you could just eat & drink your way through Portland, it's that good.