There is no "plot armor" in this Game of Thrones.
You do know the kids are adopted, right?
I'm sure they hit the door to get away from the freak show the first chance they got, and I don't blame them.
Hell, they were probably slated to be dished up in a spirit-cooking session…