I have created a searchable application for /qresearch/.
The database is filling right now. I kept only the image attachments in order to save hard disk space.
At present 52,000 of the most recent posts on qresearch are loaded in the table with the attachments. We'll see how the storage works out.
I'll advise when anons can attempt to use the system.
Searchable Qresearch
www.pavuk.com
username: qanon
password: qanon
Updated regularly with the messages and images from Qresearch general.
I'm using the 8chan JSON API endpoints. I still need to pull from the archive.json file downloaded yesterday.
My server is on a linode so I have fast response time.
You can search the text is the posts with wildcards. Say you want all posts with the word BOOM. Just enter boom.
Say you want the posts from Q with his tripcode and "boom"
Put !UW.yye1fxo in the trip code.
put boom in the comment
Click search button
voila.
Has anyone found a way to go back past the 25 pages in the console.json?
U.POSTS.NEW is the new-format table.
U.POSTS.NEW.ATT is the table of attachment for the primary table. Each one is a link to a binary
I've not been back into this thread for a while. I'm running the qresearch import process to get up-to-date. One technique that is needed is to re-scan already imported threads for posts missed during initial scans.
Threads are imported from the catalog.json file. In this state, we know the thread number and the number of messages at that time. The only time we know a thread is closed is when the number of posts >= the number in the official "bake" count.
Therefore, my program keeps testing until the posts counter >= the bake counter and then marks the thread as complete in the thread table. This then prevents re-scanning all threads because we get only the open ones.
Multiple scans of posts are needed to get all of them and to deal with duplicate threads.
I use the 8-chan post number as part of the primary key to the threads and posts tables.
8GA_1 is 8chan Great Awakening post 1
8QR_655000 is 8chan Qresearch post 655000
The big problem is going back to find threads BEFORE the last 25 pages in the catalog.json. Therefore, I can't get anything earlier than when I first wrote the import.
The import routine uses the JSON API endpoint from the boards. In the JSON is the Unix timestamp of the message. This is a native field/object type in Pavuk. Thus all timestamps are set to UTC internally.
NOW, if I could get DJT's Twitter feed in JSON, it also has UnixTime and this goes in directly.
Twitter wants me to give them all sorts of documentation before they will allow me to use their API. Frankly, I don't have the time to deal with them or the inclination.
Super simple.
Entry forms are also search forms.
Enter the data that you wish to match.
Click the search button.
Pavuk creates and then executes the appropriate query and returns the items in a Kendo grid. Scroll, resort, export to excel or click on a row to return to the entry form with your data.
searching on timestamps has issues that i need to resolve
I'm done for the day.
The comments from the JSON API include markup and JS to go to real links. This is a problem with the storage and search. I pipe the comment string through Lynx with the -dump option and this gives me clean text in STDOUT and then a separator and then the list of actual links. I put the text in the comments and the links in a multivalue table. I'll expose the links tomorrow as a separate tab in the entry form.
I can get the other boards and other threads, the issue is disk storage. Linode gives me a lot of bandwidth, but only a few gigs of disk until I change my plan with them.
The limit of an OpenQM hash file (table) is 16TB. When this becomes a problem, I can create a distributed file (table) by primary key. Say, put all 8QR in 1 portion, 8GW in another. Simply a way to have physical storage allocated
Pavuk session records are GUIDS. (don't worry, I'll purge anons out of the storage.) It was done because of commercial requirements for SOX and other audit compliance issues. Remember, I created Pavuk to build commercial apps.
The distributed file is built by using the first 2 bytes of the GUID from the primary key. Thus, it has component files:
00
01
…
FE
FF
Or 256 parts.
Theoretical table size:
256 x 16TB = 4096TB
www.openqm.com
I'm going to look at your work.
Brother Anons, I can find the IDs of the threads by using the search function on Archive.is. For example, research general #2 was post number 799. Once I know this, I can go back to 8chan and pull up the thread.
Sadly, I cannot get it with JSON. I only can get HTML. This means parsing the HTML.
This means a new string parser, but it goes into the same table as the JSON, but with more work. Here's what the posts look like in HTML
I've put out a tweet thread showing the progress and asking if someone will step up to help lead a crowdfunding campaign so I can afford a bigger Linode.
That was helpful. I would ask people in this thread to help develop the information model.
There is a "boards" table with the links to get data for each type. It can be expanded into which boards are archived where and I can automate the pulls.
I just saw this info. I need to convert my monthly plan to an hourly plan before they'll let me buy storage blocks.
Pavuk Searchable.
Linode is telling me that I can get block storage, but only by migrating my VM to the Fremont data center, getting a new IP address (SSL cert. etc.)
Crickets from followers whom I've asked to donate funds for the added expenses.
Raw video.
related