For rock, from top favorite to lesser favorite:
Academy Records in the East Village and Greenpoint (great selection of used)
Brooklyn Record Exchange (in Bushwick, large used selection)
Captured Tracks (decent, near Greenpoint Academy Records)
A1 (decent, a little rundown, in East Village)
Stranded Records in the Village (mostly new)
Forum:
Tell Me