first thought is threatened with harm to family
The WH staff working areas are probably smaller than needed and can you imagine maintaining secrecy and security in a building with public tours plus staff and visitors to a private residence.
Note the VP has a residence at Naval Observatory, but office is separate at Executive Office Building.