I love the Fort. It's straight out of Scooby-Doo. It's a nice short kayak from East End Beach here in town. I try to get out there every spring.
I'm looking at it right now out of my office window. I took this shot this April.
At low tide, the inshore part of Diamond Ledge is attached to Fort Gorges. The gravel beach between the two makes for a great landing site.
Three sides of the Fort contain gun casemates. It would have been deafening in here with the huge cannons firing.
The Fort is three stories high, with vegetated ramparts above. Apparently the ramparts are unstable, and climbing on them is technically forbidden (not to mention a dangerous proposition), although you can easily climb to their level.
Some of the inner passages of the Fort have crumbled into rubble; bring a light to explore more fully. I found ice in some of the upper levels in April 2003, but none this year. There is still a very cool octagonal spiral staircase.
The Fort is largely built of granite, and much of it is sound enough to boulder up. I explored the wooden officer's quarters located above the entrance, but found them to be very rotten and unstable.
Definitely worth a trip. It's about 0.6 miles out into the harbor. A good loop can be made including Little Diamond Island and House Island (trespassing technically prohibited), which is also the home to an old fort plus WWII fortifications).