The reality of the situation is that you can’t please everybody. Any decision for a group larger than five (if not smaller) is likely to work out poorly for at least one member. Someone will likely feel slighted, if not altogether pissed off and you’ll hear about it. Luckily, it’s not your job to make everyone happy.
Your job is to move your group forward. Your job is to ensure that as many people as possible (within reason) can participate. It’s too easy to miss an opportunity because one member isn’t able to participate, or doesn’t care for the topic or the venue or the day. Make a decision that will benefit the majority of the group and move forward.
People who want to be a part of the group will find a way to participate if its important to them and in time you’ll find the right balance to ensure as many people as possible can participate.