Why not? Because there is an issur D’oraisa of lo sechanem-Lo siten lahem matnas chinam. I know that this din d’oraisa is difficult for many people to reconcile with political correctness, but it remains the din.
There are perhaps ways to define chinam, which would allow this, though. Especially if they are neighbors with whom you have a relationship. Then it is not chinam, but rather to maintain that relationship.