They poke and poke and poke until you throw the first punch, and then "you started it", because you threw the first punch while denying everything before that.
Best to avoid the fight, as they eventually wear out. Hard though. Takes and extreme amount of patience.