The Fed is a root issue, but you're wrong about there only having been two presidents that attempted to end it. Andrew Jackson succeeded in defeating America's first central bank.
Here's is the source for the quote. I'm not going to download the PDF and look for it because whether he said it or not it is widely attributed to him.
''Gentlemen! I too have been a close observer of the doings of the Bank of the United States. I have had men watching you for a long time, and am convinced that you have used the funds of the bank to speculate in the breadstuffs of the country. When you won, you divided the profits amongst you, and when you lost, you charged it to the bank. You tell me that if I take the deposits from the bank and annul its charter I shall ruin ten thousand families. That may be true, gentlemen, but that is your sin! Should I let you go on, you will ruin fifty thousand families, and that would be my sin! You are a den of vipers and thieves. I have determined to rout you out, and by the Eternal, (bringing his fist down on the table) I will rout you out!''
From the original minutes of the Philadelphia committee of citizens sent to meet with President Jackson (February 1834), according to Andrew Jackson and the Bank of the United States (1928) by Stan V. Henkels
It's nice to exchange ideas as shilly as it is in here.
Going after banksters and killing native americans are not mutually exclusive.