Without the shedding of blood there is no forgiveness of sins (Hebrews 9:22). Cain knew what God’s required sacrifice was.
Adam and Eve had other sons and daughters (Genesis 5:4). Today, after thousands of years of genetic deterioration, it is no longer safe for close relatives to marry, but in those days there was no risk of genetic disease in the offspring of siblings.