>>520068

Code at github.com/anonsw/qtmerge does some similar things. Check it out, maybe there are some useful ideas to lift from there: anonsw.github.io

>>520179

Cool, check out qanonmap too for posts no longer retrievable. I think they have some that qcodefag doesn't have.

>>520278

github.com/qanonmap

qanonmap.github.io

not sure if thestoryofq.com is related

But they are qcodefag forks.

>>520323

Yep, but I think new ones just haven't been added yet to qcodefag.

>>536855

Good catch. I found some in my db as well.

I like the post headers in the UI. Nice and clean.

>>543389

qtmerge uses the raw JSON/HTML data where relevant from 8ch, 4plebs and trumptwitterarchive as it's source data. It also merges in the JSON from qcodefag/qanonmap. It currently uses the host, board, post timestamp and post number to sync.

I like the idea of matching the GUIDs along with a post hash using some method we agree on.

>>547789

Only HTML of archive pages.

>>550148

Let me clarify, HTML for just the archive pages (to capture threads not in catalog/threads.json). JSON for everything in else.

I'm working on how to share it, currently unoptimized and around 6 GiB of data uncompressed.

>>553109

For now it just uses a dedicated file system.

With images gathered so far this mirror's total size is 193 GiB.

>>560076

Yes, unoptimized and incomplete.

>>568187

I did some research on collecting the CBTS threads from 4chan/pol the other night and the results might be useful for others. They can be found at the bottom of the page here:

https:// anonsw.github.io/qtmerge/catalog.html

It's still a work in progress.

>>569170

Were you able to capture any of the original 4chan JSON/HTML data? I wasn't researching Q at that time so I've relied on 4plebs.

>>569793

Excellent. Will that raw JSON data be in the DB as well?

>>569900

I did notice, those are great ideas. Can I suggest letting each user have their own copy/edits of the metadata? The user-specific data could then feedback into the system for suggestions to others, etc. But primarily it gives the user some way to control the interference/noise.

>>570566

4chan JSON for pol between 2017-10-30 and 2017-12-01.

>>570660

Thanks, 4plebs is good for now, but a second witness is preferable. Started archiving Feb 15, but some old data was still available at the time.

For 8ch these are the oldest breads I have:

pol: 10509790 (2017-08-28)

cbts: 10 (2017-11-21)

thestorm: 1 (2018-01-31)

I don't have all breads after though, it is incomplete.

I've since stopped archiving pol/cbts/thestorm to save time/space.

>>570809

Not enough due to VPNs, DHCP, etc. The login may be the best way.

>>570983

I might already have it, is it in the QJsonArchive.zip from earlier?

>>571161

I do, I'll call your dataset QCodeFagNet unless you want a different name. Instead of the zip I'll pull it from your github.

>>571310

Should I hit _allQPosts.json?

Below is the qtmerge modified raw dataset (text-only) as of 2018-03-14 02:07 UTC.

I'm putting this out in the hopes that it may be useful to others for ETL, mining, search tools, archiving etc.

Some notes:

• The data is a synthesis of the the qtmerge datasets: https:// anonsw.github.io/qtmerge/datasets.html

• For an idea of threads that are available see: https:// anonsw.github.io/qtmerge/catalog.html

• eventcache.json file contains the posts/tweets/etc in chronological order. The type attribute currently dictates the local object structure (working to fix this to be more clean)

• refcache.json contains the detected post cross references (this is a work in progress)

• The referenceID attribute is the "primary key" between the files

• Timestamps are Unix Time and time strings are US Eastern

Extracted size: ~850 MiB

SHA-256 sum: d6ed89da05c0b714fc66b04ca66a8d701456d882d5f128ee1cef26c8d2e22eb6

http:// anonfile.com/dazfO8d4ba/qtmerge-text-2018-03-15_05.18.37.tar.bz2