Hello and welcome to Binky Bunny!
There is a chapter of the House Rabbit Society in NYC. http://www.rabbitcare.org/index.htm
They have a list of sources of hay, but it looks like they are all mail order companies, which are not going to be very cheap. I get my hay from drsfostersmith.com, who often has sales and they normally have free shipping when you buy at least $19 worth of items. Sometimes they have free shipping no matter what amount you are spending. I get Oxbow pellets and hay, and American Pet Diner hay from there. They have other brands too.
I don’t know how far away you would have to travel to find a farm or place that sells bales of horse hay. That’s a lot more economical than buying hay in bags. But most of us don’t have access to hay farms.
The cheapest litter pellets will be either horse stall pellets, such as Equine Pine, or wood stove pellets (with no accelerants). Those both come in large bags, around 40 pounds for only about $6.00. They can be found at feed stores, horse supply stores, big box stores like Home Depot, hardware stores or Tractor Supply. I don’t know if any of those stores are in or near NYC.
You can contact that HRS chapter and ask if they have any activities for rabbit or places to meet other owners. They do have a list of rabbit savvy vets on their website.