>I have to go in every couple months and clean up duplicates and remove moldy bread notes
Hash the entire post and, if practical, all of the images attached to the post. The hash can then be used to drop the dups, during ingestion, and prior to indexing.