One of my friends suggested that maybe if Yaxley was doing side-by-side apparition, they would be in the house even if they wouldn’t know where they were, but if Harry would appear there, they would see him.
(Voldemort could look through the window because the secret being hidden was where the Potters were hiding. Here, it is just the location of Headquarters that is being hidden, so they could get in, and simply wait for anybody to come.)