It’s not about putting people in their place.
99.9% of normal parents know not to come visit on their own without calling at night. You can safely assume that if these parents are doing this to their kids, they lack a certain basic sense of privacy and proper behavior.
People who lack a basic sense of propriety need to be told politely and firmly what the limits are, and then you have to stick to them.