That's most true for the soft cartilage in and around the inner ear. Some cosmetic ear changes are possible.
Read about it from the forensic scientists who looked into a famous celebrity replacement hoax from the 60's expecting to debunk it in a few minutes and, instead, spent months coming to the conclusion that indeed Paul is Dead:
https://plasticmacca.blogspot.com/2010/01/forensic-science-proves-paul-was.html
http://plasticmacca.blogspot.com/2009/12/fauls-false-ears.html