Christ didn't tell Peter He was building the church on him. He said you are Peter (stone) and on this Rock I build my church.
The bible tells you multiple times God is the Rock.
Those who claim apostolic succession from Peter (you know those false apostles mentioned in Rev 2:2) are the ones teaching Peter is the rock.
Read revelation, the stone the size of a great millstone is thrown down.
"And a mighty angel took up a stone like a great millstone, and cast it into the sea, saying, Thus with violence shall that great city Babylon be thrown down, and shall be found no more at all." - Revelation 18:21
Don't worry, Peter will feed the sheep.