It sounds like maybe the whole house belongs to you and you are just selling the chometz in the house, so I guess that’s why you can be there.
But I always thought that when people sell their sefarim (technically the chometz in the sefarim), they are supposed to try to avoid using the Sefarim or something (although I never really understood what that meant – either they should be allowed to or not allowed to.) Basically, these halachos are confusing and I never really understood how any of this works, and there are probably lots of opinions and lots of different situations, so just ask your LOR and do what he says.
I’m very happy you asked him the main questions meanwhile and got the answers.