I think I've recovered enough from summer's events to get caught up (if that's possible). I'm working on downloading stuff right now. Next, I broke something in my local database update system about a month ago, and I'll have to fix it before I can update my databases. But that shouldn't take more than a day or so. After I get my downloads caught up and the database is updating, I think it's time to get things set up to create posts for the front end site. The front end is more user friendly for someone who isn't as familiar with Q. The back end research database is for the regular anons more than anyone else. It was originally created to support my own efforts in creating the front end, and then I decided to share it.
PageCap Army
I'm really struggling to keep up with the work in developing my site. For a while, I put focus on creating a research database others could use. But the real work needs to be in building the front end of my site so that normies can refer to it.
There are a couple of features that make the vision I have for my site different.
-
I want to make context available for Q posts, both backward and forward. So far, my tools are only developed far enough to create backward context.
-
I want to have page capture archiving of at least the cited articles. Not screen, but the full page. (Videos are a bit too big to be keeping archived copies of them all.)
Getting and processing the screen captures is taking too much of my time. I have been wanting to find ways to delegate some of the work of aggregating the information, and I think I have come up with a way I can do it without sharing and exposing my development database. I could really use a PageCap Army to create the page captures we need.
Here's my vision for it:
A page would be created either here in qresearch or in a new board for the purpose. When a Q or notable post comes through with a link to an outside article, the request could be made on the PageCap Army thread. Fellow autist anons could then do the page capture using a tool such as FireShot to get the whole thing in a single image.
Then, what I've been doing with these, is cleaning out ads and links to related materials, compressing the white space, etc. I keep the header and a compressed footer and just the article itself. Sometimes multiple articles come up on the same page, and I'll crop those off, too. This process has been taking up a lot of my time, and this is an area where I could use some help. Right now, my time is better spent building the tools needed to get the completed posts for my site up there with that forward and backward contexting. I got so wrapped up in supporting the Research Tool stuff that I couldn't move that forward. And I need to do that. It's more important, I think, than keeping the online Research Tool up to date. It didn't help that I had to stop what I was doing for a couple months to take care of a family matter. So now I'm working to catch up.
I like PhotoShop for working with page captures because it's easier to use for the compressing of the images. Specifically, a select can be done across the edges of the image. With Paint, it's more difficult to do selects because the select must begin on the image itself. Sometimes, though, the page capture files are too big for PhotoShop to handle. Is there another image editing program that can handle the larger page capture files? The ability to scroll a select is key since I often begin my selections toward the top of the image and need to get everything below it all the way to the bottom into the select. Plus, the layering features of PhotoShop have proved helpful, too, when getting the content compressed, especially when the page captures get messed up by floating elements on the page and cropped screen caps must be pasted into place to repair the page capture.
Maybe I need to post a before and after image of the work I do to compress an image so people will know what I mean by it. Next time I come across one that needs extensive work, I can do that.
The focus for the PageCap work is Q posts, posts in the backward context of Q posts, and notables and their backward contexts, especially if they link to Q posts (which will create the forward context of Q posts).
The Research Tool system was originally built to support the curating of the front end site, and that's what I really need to be doing. Since no one piped up and said that the online Research Tool was important to them (or would be if it was up to date), I am shifting my focus. I won't be supporting the online Research Tool anymore, and I may even pull it down entirely and free up space for the posts of the front end site. My focus will be on putting together a site that can help normies understand what is going on.
While I'm at it, is there a good article on how to have two MySQL databases open at the same time, one on the local server and the other on a remote server? Being able to do that would really be helpful for speeding up the update of the online site.
qanon.pub does backward contexting one level deep. So if a Q post has a link in it to another post, the qanon.pub site will show that post together with the post it calls. My research system is currently able to show context as far as it will go. Forward contexting shows those posts that link to that particular post. Obviously, I don't want to show EVERY post that links to a Q post. That is why I am currently working on marking the notables and creating the backward context chains for them. When a notable's context chain includes a Q post, then I want to include that as a forward context chain with the Q post. That is where I'm at in my system development. I have the data (mostly). The work is in getting things linked up. And also, I still have to build the piece that automatically creates WP posts out of the selected context chains. I think I've got that partially developed, but it still needs work.
Oh, and another thing: If I can recognize that a notable relates to a Q post but the context does not already specifically include it, I already have the capability of noting that with the post and thus forcing the linking of the notable to the Q post. But that takes human eyes to recognize.
Here's an example of what I'm doing with the page captures. Q was driving me nuts when he was posting these. Getting the page caps prepared for a single post was taking up the better part of a day.
http://localhost/q-questions/research-tool.php/type/post/site/8ch/board/patriotsfight/post/80
Wait. You need the remote version of that.
http://q-questions.info/research-tool.php/type/post/site/8ch/board/patriotsfight/post/80
I notice that this post is not up there. But I'll put it up so you can all see it. Give me a few minutes. I'll let you know.
It was up there. I just hadn't converted the 8ch to its alternative. (I was told early on not to expose 8ch, which I don't. How does qanon.pub get away with it?)
Which looks like this:
http://q-questions.info/research-tool.php/type/post/site/NewChan/board/patriotsfight/post/80
Here's an example of a post with backward context.
http://q-questions.info/research-tool.php?single=250018032640
One of the posts in the context chain shows how I force context. The missing image in the post is a screen cap of the Q post listed before it in the chain. I do have the image, but the image syncing system isn't working all that well yet.
This is an example of what it is my goal to produce. The post that shows in the index is the pink one. The backward context is above it. The forward context is below it. Given that there can be multiple forward contexts, I will probably implement the forward context more the way the contexts are implemented in the Research Tool, with the divs that can be shown and hidden, with a box around the displayed div. Each forward context will have its own box.
http://q-questions.info/2017/12/05/cbts-general-38-we-are-the-storm-no-34663/
This is what the post looks like on my development machine. It looks rather awesome when all of the images are present.
If you'd like an example of why I'm fixing page caps, check out this site:
https://www.dailymail.co.uk/news/article-2211092/Scandal-hit-G4S-warned-employ-security-guard-murdered-colleagues-Iraq.html
This is the result. Much smaller, less trash.
It's close to 6000 breads now. Quite a lot. It still fits on my drive. I'm amazed.
You'd be surprised how much clearer stuff gets with the larger context. That's why it's my goal to provide it.
And the page caps: "Archive everything offline."
That's a good question. I've been doing it manually. I'm sure it can be automated. If I were to do it automatically, the file_get_contents function can open a page. I would use the DOMDocument to parse the page. I haven't looked, but the various boxes on the catalog page are probably divs. Then it's a matter of keeping track of the responses for each one. If the count goes up, go and poll that page for the new posts.
When I'm working on this stuff, I'm quite capable of getting so in the the flow of it that nothing else here gets done until it absolutely has to be done. I get so locked in when I'm working on it that work/life balance goes out the door. Not good, really. When I'm not in the flow (which is right now), I kinda flop around a bit. It's partly because I'm still too fatigued from pulling a near all-nighter a couple nights ago and then waking up after too little sleep. Looking at my project, I think the biggest bang for the buck addition to the code would be to automate the WP post creation, complete with the local build, the remote copy, and the image ftp. All Trump tweets and Q posts would be automatically included, plus any other specifically marked posts such as map posts. For now, the posts can be built with on the fly backward contexts. Later, as the processing code develops, the posts can be updated to include the forward contexts that answer Q's questions.
Where is everyone at on time zones? We talked about this early on, and the consensus then was for GMT. So I've got everything saved in GMT, and that's how I currently display it. But I'm also seeing a leaning to Eastern. SerialBrain2 uses Eastern in his gematria work. It would be a trivial thing to produce the final posts in Eastern, if that serves us better.
>Straight page scraping is a mega pain in the ass. I'm not interested in that. All the info I need is available in the JSON and it's easy to get to. https://8ch.net/qresearch/res/2352371.json
Yes, that is much easier to read. No parsing necessary. It's nice to have that available. But the parser's already built. So for now, I'll leave it.
How are you retrieving the images when you're using JSON?
This looks like it's in the genre of the actual workings of the Internet itself. Interesting, but it probably isn't anything I would be working with personally.
Have you come up with a good algorithm for those notables? I've got one that makes preliminary identification, but I have to make the final determination. Occasionally it gets it right. More often, I have to make some adjustments.
That's similar to what I do. But because of the way I create context for things, I need to mark the preliminary ones that are posted toward the end of the page as well. For that, I'll look for a certain count of posts by anyone plus the word "notable". If one of those later notables gets caught up in a context chain, it can really mess things up.
To clarify, those are "OR" conditions, not "AND" conditions. Not all bakers put "notable" in their preliminaries.
Don't forget to stop a crawl at a notable. Otherwise, yes, they could get quite huge.
What was in your text file? I've been doing the crawl page by page. It's fairly hands-on because I have to be more careful about shill images, etc. when I'm creating a one-to-many table of the originating post with the post numbers of its backward context.
Seems to me, most of what isn't "notable" is either Q (different type of notable) or on another board somewhere. So yes, that should be close.
If you're concerned about the size for some reason, maybe you can abbreviate a bit. If you know everything is at https://qanon.news/api/bread, you can leave that part off and concatenate later.
My host has been inaccessible since Friday night. If I buy new hosting (and I'm leaning that way), I won't be putting up the Research Tool on the new host–just the WP front end.
Everyone's got their own project, it seems.