If we are saved we DO leave our guilt behind. Kindly read Romans 8:1-4:
1Therefore, there is now no condemnation for those who are in Christ Jesus, 2because through Christ Jesus the law of the Spirit who gives life has set you a free from the law of sin and death. 3For what the law was powerless to do because it was weakened by the flesh, b God did by sending his own Son in the likeness of sinful flesh to be a sin offering. c And so he condemned sin in the flesh, 4in order that the righteous requirement of the law might be fully met in us, who do not live according to the flesh but according to the Spirit.