Would you allow someone to just walk into your house and sit on your couch?
I am thinking you would say NO!
Society currently has a hold on you though that it is EXPECTED of you that your dog is public property and should be approached and touched by anyone to do with what they want, even over your objections as the dog owner.
So just change the dog petting thing into the personal space portion of your boundary setting and it will get much easier.