You are soooo not alone. Thank goodness for bdw! I have the same fears that this will happen! As plane tickets become available, I'm trying not to be a bridezilla about nagging people about booking!
Looking on the bright side, those who will be there will have a great time. It'll be intimate and wonderful.
Sharing the day with more friends & family may be nice, but it's good not to have anyone resent you for the cost, timing, etc. I've tried to keep the perspective that while there are some people, I REALLY want to come, I also want them to come for themselves, not just the wedding.
And you are totally RiGHT, this is not for everyone else. It's for you and your FI. They are tons of fun wedding things to focus on aside from your guest list! This will be a happy event!!
sidebar: My FI was disappointed that his aunt & uncle wouldn't be able to come to ours because of health issues. To still share our wedding with them, we're going to head to LA to have them at our legal/city hall ceremony and a small celebratory dinner. Just an idea for a way to celebrate with those closest to you who can't make it.