They don't have natural kids. My guess is there is a fertility problem. They decide to adopt, want a certain ethnicity, but that particular ethnicity means a long wait time when done legally. Enter Epstein.
The adoption issue, AFAIK, was insinuated by the NYT in 2005 (confirmation hearing), then raised again by NYT around Obamacare debate time. That fueled the comp'ed rumors.
My guess, again, is that it's a very emotional subject for the Roberts, especially the wife, due to an inability to have natural kids, and the fear of having the kids removed was too great. I'm talking Obama administration, here.