Try not to get too bummed out. Those who answer right away are pretty likely to say no because it's very clear that they just can't make it, for any number of reasons. You will get to the yeses , some might be waiting for prices, or being able to get those days off work, or crunching numbers to see if they can make it and they will respond, just not as quickly as the flat out "no's".
I was getting pretty bummed out when some people said no too (those were the first responses we got) and I thought "no one will come!" but as it turns out, most said yes and booked with our TA and everything went quite smoothly. Then I started reframing people not coming, I thought about the money we'd save by having less people! (I feel awfully cheap saying that but hey, it was the only positive side I could find! haha)
Like others have mentioned, at least you're getting definitive answers, many of us have people who are still stringing us along, not sure if they're coming or not, which push you to leave a big chunk of the planning until the last minute.
When it comes to invitations it's such a touchy subject. I sent everyone invitations, to my fiance's disapproval, even if they said no. When I gave it to them I told them that I knew they said no but I know some people like to keep invitations so I was giving them one anyway. that way they didn't feel pressured but they also didn't felt let out. Some people get offended if they don't get an invite (even after declining) and others don't care so it's kind of a guessing game there. I played it on the safe side but I know many brides don't send invites to the no's and everyone was cool with that.