Jamali, on 28th Street just east of 7th Avenue, is a good place to start. That whole block has plenty of plant stores. Planter Resource, nearby, is also excellent for pots and planters.
The Greenmarket in Union Square, especially in spring, has a nice selection of herbs, perennials, and annual flowers. Also try Chelsea Garden Center (having a sale this weekend), and while there stop in Chelsea Market (on Ninth Avenue between 15th and 16th Street) to check out the food and architecture.
Gowanus Nursery in Brooklyn is easy to get to by subway and has a great selection and staff. Also check out the plant stores at the Brooklyn and Bronx Botanic(al) Gardens.
I don't live in your area, but I'd suggest that you also get in touch with some Master Gardeners from your area and get the low down on where they like to shop. Many of them are addicted to gardening
