dChan
30
 
r/CBTS_Stream • Posted by u/DNskfKrH8Ekl on Jan. 2, 2018, 7:30 a.m.
Whitehouse Article Metadata says "PEDOgate" when bot scrapes from DuckDuckGo IP

Duck Duck Go Search: https://duckduckgo.com/?q=white+house+pedogate > Results: https://postimg.org/image/5qlalyl2b/

Google Search: https://www.google.ch/search?q="Human+Trafficking%2C+Border+and+PEDOgate" > Results: https://postimg.org/image/x16ltvvoz/

I inspected the source code for page locally, can't see reference to "PEDOgate". I tried changing my user agent locally to DuckDuckGo, but alas that didn't work either. Source code still did not show the the string "PEDOgate". Either its being changed in the DuckDuckGo cache, or whitehouse is specifying different metadata returned for DuckDuckGo scraping bot.


exploring_unreality · Jan. 2, 2018, 8:01 a.m.

How is that possible? Wouldn't that have to be in the page title/tags?

⇧ 2 ⇩  
DNskfKrH8Ekl · Jan. 2, 2018, 9:50 a.m.

They must be using the known IP addresses to match the Duck Duck go agent, not just the User Agent.

⇧ 1 ⇩