Trump may not hold all the cards here.
It is possible he does not want to deal with the optics of a building thal is more than half empty.
Possibly that he has no legal basis for even appearing on the podium - Pelosi could schedule other events in there.