I am having the same problem. I know working in an office it can be difficult but in all honesty if you don't want them there than don't send an invite. If anything, maybe co-workers will throw you a shower and they can attend that instead.
Everyone understands nowadays that money doesn't grow on trees and if you could you would invite everyone but there has to be a limit somewhere.
Hope you get your answer