Fixing the notables updates on the site today
Also happy to add links to other archives for posts/threads/videos/whatever, just need some info about how they are formatted
actually it was the server running out of disk space this time.
catching up now.
>Would it already help if I created a General Notable thread and updated it daily with all notables?
yes, if you can do this, definitely! the moar notables archives the better.
>What works for me all the time is this API:
>You can also implement the wayback machine URL lookup code in JavaScript, so that the client side checks for the archived data.
>Or you could do it on demand, like when a user clicks on a video the first time, you look up the URL. If you get an URL, you save the URL in your local database.
We will look into this, I like the idea of the site trying media.8kun.top first and then other archives if that's not available any more.
>any chance of providing a FORK of your noteable harvesting script/source
We will likely be releasing the code at some point soon. Lot to do first, basically need to take a good look through all of it and make sure there is no sensitive info that shouldn't be there.
>When I'm done with my collection code, I can either put the exact logic I'm using on here, or even put the actual code on here.
Excellent, the more the better. Notables and in general better sources of news are too important for us to be relying on a single site or service.
Likewise anon. It looks like these archives are only for /qresearch/, is that correct? wearethene.ws also has bread archives but they are not linked from anywhere at the moment: https://wearethene.ws/qresearch/res/10249855.html#10250524
>Heaven is watching.
Heaven is watching all anons, bakers and others who help to fight for a better world. None of us can do this alone.
Thanks everyone for the support, we really appreciate it.
for parsing JSON it's best to use a library dedicated to that task because there are other escape characters too, and "\uXXXX" "\uXXXX" "\\uXXXX" etc all need to be handled differently which has to be done using a parser. no idea if SAP has a full JSON library, all our stuff is based on linux.
there are two reasons wearethene.ws doesn't store videos and PDFs. first one is disk space, and second is safety, those are more complicated file formats that can contain all kinds of nasty stuff (especially PDF).
and I don't trust wayback machine either, they definitely delete stuff. I do have a local archive of most media posted on the boards that could be uploaded somewhere but that would take up a lot of space.
Problem?
wearethene.ws is catching up now, should be up to date within a few hours.
This time, the notables parser saw a bunch of changes in the structure of the dough and stopped at bread #13698. The changes were related to fighting over placement of warroom links. The parser looks at a few things to try to detect shenanigans in the kitchen and flag the resulting breads for manual review…… looks like it worked.
And I've been busy with IRL stuff so couldn't get it properly fixed until now.
>Notable thread is now up-to-date.
>>>>/qresearch/7003045
This is awesome, thank you AAA. More independent alternatives to wearethene.ws is a very good thing for everyone, because right now wearethene.ws is a single point of failure for a lot of anons and people they know. I think the dough should be updated to mention that this thread is being kept up to date now.
I very much appreciate the support anons, and sorry for any disruption caused. You fags will live though.
p.s. No one involved with wearethene.ws has been doxxed, like a few people were theorizing on the main board. Sucks for the guy behind qmap.pub, but making facebook posts about your high-traffic Q-related sites is pretty bad opsec. None of that here because we operate our stuff from behind seven proxies.
glad you are enjoying doc.
bakers should be able to change the headers in the dough as normal. I relaxed those checks a bit and I will also be looking in on that more often to make sure it works right.
>Can't you parse threads in a way that threads that are marked for manual review are skipped at first, so that at least other breads are shown?
I'm sure that can be improved further, but one problem with this is it's very hard for automated code to tell the difference between these two situations:
-
#15001 is valid, #15002 through #15010 look different, #15111 is valid again
-
#15001 is valid, baker makes a mistake and bread number skips to #15011
I think the best solution for now is to add some more monitoring for what is happening with new breads.
>Is it such a problem if there is a duplicate thread, or a thread is missing?
Yes, one of the goals of wearethene.ws is for the record of notables and bread numbers to be as good as possible, so that wearethene.ws serves as a high-quality data source for people researching these events in the future. Different aggregators having different goals is also fine.
The problem with waiting for BO/BV actions is the delay it would introduce, first of all, and second that BO/BVs usually don't fix all the issues with bread numbering.
>Over the last months I have never really seen that happening (of course maybe I just missed it).
Maybe, but what does happen a lot is stuff like bakers accidentally putting the wrong bread number in the thread title or the notable buns. When the bread number is baked as #12800 but the correct bread number should be really #13800, notables from a thousand breads ago could easily get overwritten.
This used to happen earlier in the site's evolution. It was a huge pain to clean up when it happens accidentally, and it would also be easily exploitable by bad actors to hide specific information.
This is why the parsing for wearethene.ws is pretty strict now. Usually this works out fine, since part of the baking process is fixing those mistakes when they're noticed, but in any case the parsing will continue to evolve over time.
>Can you please change video URLs to stop going to invidio.us, because that instance is down, and you get a selection which includes jewtube.com, which is kinda ridiculous.
if the instance list stays up at invidio.us, this is better, because then we aren't sending all our traffic to one instance.
and I just see yewtu.be which is fine. anyway if you don't wanna see the words jew, joo, muhjoo, etc, then chans are not the right place for joo, kek.
>use the thread number, not the bread # for saving notables.
wearethene.ws uses both. bread numbers are for humans, thread/post IDs are for computers. it's more user-friendly to have sequential bread numbers but it does make parsing harder.
>recheck if the bread you took it from gets deleted or not and use it straight away
most duplicate breads never get deleted. they usually get filled or locked.
>isitwetyet … replied they would take care of the JSON data issues
excellent, you're a lucky anon ;)
>Recently discovered wearethene.ws, an inspired piece of work; dark mode awesome! o7
glad you are enjoying anon, o7
>Is there a mode so links open in a new tab?
I guess this would be for links to other sites? at the moment this isn't an option but we can probably add it. for now just remember to right-click or long-press on links.
>codefag reporting for duty if needed.
you or other codeanons still here? leave a protonmail address in a comment here, and we'll get in touch.
>looks like wearethenews last grabbed qrb #59.
thanks for the reminder fren, I'll get the qrb notables up to date now.
Jumping straight to FUCKERY any time something happens is not good research methodology. Credits ran out with the hosting provider for wearethene.ws and qanonbin.com, paid and sites are back up now. They are charging a bit more than necessary but have been perfectly trustworthy other than that.
>Possibly even multiple (and different) notables posts.
There are a couple of threads to fill in but not too bad. Anyway this is why notables parsing is "strict". The site will basically just pick one of the posts that's formatted correctly, but I will look at the notables thread later and see what should be combined. I don't have time to be doing that constantly though.
I've read a bit about Uribe, seems like he is a piece of shit who needs to hang. Good luck fren. https://en.wikipedia.org/wiki/%22False_positives%22_scandal
Next time you should post in the main thread, see >>11704 for instructions on how to get there. Also don't post your name or email on this site.
Message to Q doxxers attached.
>Decide on some checks and balances bullshit and get everything squared away please.
Get on the general board and help "square it away" anon. I don't have time to properly deal with the massive amounts of faggotry on the board this week.
>Makes me think that operators of wearethene.ws are part of a side of that battle versus doing what’s best for all anons and normies
No, we just don't have unlimited amounts of time to deal with this childish bullshit, and fixing it properly requires going through the notables for every single bread over the past week or so.
In the past, anons and bakers have mostly been able to curate notables pretty well, but since this started that hasn't been true.
>It seems wearethene.ws is comped as much as the bakers on the main board.
See above, and then kindly fuck off.
Want to send a HUGE thanks to AutoArchiveAnon for compiling the notables over in the /qresearch/ notables thread. That is going to make it much easier to get wearethene.ws back up to date under the current situation. Now that is what you call using the whole buffalo!
Working on it. Thank you all for your support.
>I have no idea what goes on over at the boards
"What goes on over at the boards" can be thought of as a training exercise for real-life government.
If you don't like what you are seeing, you need to get involved and help fix it.
I concur with your analysis anon. "Attacks will only intensify."
Go fuck yourself.
Your post really helped!
Gracias amigo. I will continue doing what I can.
The site will be up when we have a few hours to deal with this again, it is somewhere between a half day and a full day of work to properly clean up after a faggotry outbreak on the boards. It's a bit like disinfecting everything for covid but online.
If you want wearethene.ws to keep updating immediately 100% of the time then the best thing you can do is get on the boards and make sure BO, BVs, bakers are doing what they are supposed to be doing.
Other, new aggregator sites that do not rely on wearethene.ws are also very welcome. "Diversity is strength" as the globohomos say but in this case it's actually true.
>i'ma bakin' over on /midnightrider/ board right now.
Thanks for creating a refuge. I'm curious why you guys chose to create a new board instead of using /qrb/?
Anyway, this will take some work to add the new board and get the split notables parsed correctly. We will do that but getting qresearch stable again is the highest priority (it's catching up now).
>Would be good to know get a LIST of what formatting problems will throw off wearethene.ws
This specific information would immediately be exploited by shills. General rules of thumb - just follow the same standards as past bakers, only change the dough calmly with anon consensus, and try not to fuck up the bread numbers or split the breads too much.
---
Thanks to some of the other commenters here for your patience, you know who you are.
---
Then we have some ungrateful assholes who are just coming here to complain about something that is being provided 100% for free.
It's not a pissing match either, I literally just have too much shit to do to be babysitting notables, and the garbage that has been coming through the boards lately is so bad that the site stops, as it is designed to do.
Here's a couple of better options for next time that happens:
-
Go help fix the problem at the source. Learn how the main board works and help kick the shitheads out. Q is trying to teach us that decent government is only possible if we get involved directly...... and apparently some of you are slow learners.
-
Go over to >>>/qresearch/7003045 and read the notables there.
-
Learn to code and make your own aggregator site. We will happily add it to wearethene.ws/about unless it's a total piece of shit, like some of the comments here.
Options.
-
Sure, we can close the website down.
-
Or you can learn to code and build a better alternative yourself.
-
Or you can enjoy what you have been given for free, using an enormous amount of time that could have been spent doing other things.
Your pick.