I believe former presidents and their spouses are issued diplomatic passports for life. Revoking this would likely be complicated and very public and could only be done by the State Department, a Federal Judge, or The president.
Trying to find the law/code that applies, but its probably relevant to BHO and HRC travel. I know this was an issue discussed after Nixon's resignation and his travel to china.