New Threads database works with search. I searched for boom in the comments and it found those threads.
I had to wash the HTML through Lynx to clean it up. The [nn] are from lynx.
The application system is my own. It's inside of an OpenQM postrelational database. You operate from the web app provided.
I'm pulling in the posts from the research board. Only the general threads and not everything.
Image attachments are pulled in as well as thumbnails.
Also, only back as far as bread 474 or so because that's all that's available from the JSON API endpoint's catalog.
over 100k posts thus far.
There is some crapeau in the posts that I need to run through the Lynx filter. Proved the concept with the threads import.
***
step 1: import catalog
step 2: import thread headers + clean
step 3: import posts + clean
will advise when ready. searches were failing because of odd characters in the data…
There is also a People database where anons can add people to the database system and then link into posts. Need to add some fields like alias names and such.
So, if you see HRC in a post, you can search for her and link her out of the people file. This improves the data model and makes it easier for others to search.