Hear me out.
"I know, I know, ugh, sorry, I'm just not going to have a press secretary is all. If you want to know what's happening, you'll have to go to my rallies. Many great people there. Some might say the best people. I'm just not going to have a press secretary is all, haha."
It would still cause seething and merriment.