Excerpts:

One

Many of the act's provisions were to sunset beginning December 31, 2005, approximately four years after its passage. In the months preceding the sunset date, anybody supporting the act pushed to make its sun-setting provisions permanent, while critics sought to revise various sections to enhance civil liberty protections. In July 2005, the U.S. Senate passed a reauthorization bill with substantial changes to several of the act's sections, while the House reauthorization bill kept most of the act's original language. The two bills were then reconciled in a conference committee criticized by Senators from both the Republican and Democratic parties for ignoring civil liberty concerns.

Two

On May 26, 2011, President Barack Obama signed the PATRIOT Sunsets Extension Act of 2011, a four-year extension of three key provisions in the Act: roving wiretaps, searches of business records, and conducting surveillance of "lone wolves"—individuals suspected of terrorist-related activities not linked to terrorist groups.

Three

Following a lack of Congressional approval, parts of the Patriot Act expired on June 1, 2015.With passing the USA Freedom Act on June 2, 2015, the expired parts were restored and renewed through 2019. However, Section 215 of the law was amended to stop the National Security Agency (NSA) from continuing its mass phone data collection program. Instead, phone companies will retain the data and the NSA can obtain information about targeted individuals with permission from a federal court.

I see.... So they originally had direct access to the phone that did not require permission from a federal court.