For Queens, I definitely agree with you. Catching the 4 at Grand Central, 51st Street, or 59th Street, depending on which line you're taking into Manhattan, is exponentially more appealing than trekking to Jersey. Where in Brooklyn did you live? Depending on where, a trip to the Bronx would be equally as daunting as a trip to Harrison, in my eyes.
I was in Prospect Heights, so it was a short walk and a 4 ride uptown. I did that twice for derbies at NYCFC, and it was a breeze.
In theory, the commute times would not be terribly different (any of the subways to Fulton, Path to Harrison), but when you factor in weekend schedules for subways and PATH, it can be quite the hassle. A number of people I've invited to free games with me have flatly turned down weekend games (some refusals were a function of euro snobbery, but not all). Weekday games from FiDi are a much easier sell, but also relatively rare.