Bot/spam? These two threads seem vaguely similar in content:
"I have followed Q since 11/2017 and still can't figure out your post/thread protocol."
https://8ch.net/qresearch/res/1708306.html
"Been following Q since 11/2017 and still can't figure it out. How do all the shills make it but little ole me can't? Love the anon fags anyway. "
https://8ch.net/qresearch/res/1708422.html
Recommend delete/purge.
>Duplicate post must have been deleted. Copypasta is common in 8chan, unfortunately there is no way to effectively stop duplicate posts without compromising the entire board.
When you say compromise, how do you mean?
Depending on the issue, it can be solved.
Store posts (no need to store IDs) in 'unique post' database table.
Look up against unique post (this needn't impact user experience: they can post, but the system still runs the query and auto-alerts if a match is found).
If you have something like millions of posts that are stupid lengths long, then:
1) Short hash look-up (EG something crude like SHA)
2) If a short hash matches, do a direct post comparison (so store hash + full post)
So if a post's hash matches that found in the database of other post hashes, it triggers a more in-depth comparison.
Sort the hashes in alphanumeric order (insert in alphanumeric order).
Keep position indexes for the first two letters. EG A2 is position 1000123.
When you calculate the post hash, look up the first two letters for position index (getting the first position index for A2, and then the second, A3).
Perform a binary tree search down that position index (like dewey decibel system in libraries) - so start halfway, if your alphanumeric sum comes earlier, halve the next lot (maximum of 7 hops).
If found, full post comparison, if match, deny/flag.
If not found, add.
Done right, you have a multi-tiered rapid post lookup system:
1) Shortened hash (fewer characters to compare in a string lookup, and also serves as post validation in-case of tampering)
2) Two-letter positional index system
3) Binary tree search
4) Direct comparison (for accuracy)
It might take a bit of effort to implement at first.
If you wanted to be super lazy, just make the first two letter alphanumeric lookup a dedicated table (AA through to 99). If the table is missing, so is any entry and you can create a new table.
Feel free to steal the ideas, or if there's a specific problem, let me know and we'll work something out.