Apologies for an angry cross-post mastodon.social/@khobochka/113…

There are some suggestions there, ranging from easier to more complex than redirecting to generated content, e.g. redirecting to Hetzner's speedtest file, setting up tarpits for bots and a few block lists.
There's also a list of AI fuckery here, a few hops from which people confirm that 20-75% (depending on amount of content on the site) of traffic is LLM crawlers and they outrank the usual Wordpress attacks during on-calls.

None of this, however, in any way compensates the sad reality that this would eat away your time and compute, simply because the LLM training infra exists, this makes me absolutely livid.

GitHub - msigley/PHP-HTTP-Tarpit: Confuse and waste bot scanners time.

Confuse and waste bot scanners time. Contribute to msigley/PHP-HTTP-Tarpit development by creating an account on GitHub.

^GitHub

Sorry to read this. Hope there's a way to mitigate this.

I have to be cynical, but I find it pretty rich that these companies, as well as the techbro scene in general, will espouse meritocracy on every end, yet they will happily syphon the work and hobby time of open source and Fediverse enthusiasts without explicitly giving back. But maybe I'm also naive and don't realise that they support Fediverse infrastructure/coding. Correct me if I'm wrong.

for some reason, this post went semi-viral on mastodon and hackernews, and I now have a fun combination of

1. people who question my experience because the current robots.txt of the wiki or some arbitrary archived version did not contain any relevant blocks (despite me not specifying when or how I blocked them),
2. people who made the exact same experiences before,
3. people offering "suggestions", despite me not asking for any - and a lot of those "suggestions" either don't make any sense, or involve hosting an LLM myself to generate nonsense for LLMs to consume, while also wasting another fuckton of resources.

I love the internet.

Um, what did you do?

I turned off the public history pages.

People giving unsolicited advice are not being helpful, just annoying.

They are a scourge.

Looks like Dennis went viral in the activitPub space thx to friendica …

no. it was primarily someone taking a screenshot and posting it. someone who took a screenshot of.. diaspora. while being logged into their geraspora account.

but of course it's a friendica user who also sees nothing wrong about posting unsolicited advice who is making wrong claims.

@utopiArte Your grasp of human psychology, internet culture, and science in general, is weak.

Consider staying off the internet.

(How'd you like that unsolicited advice?)

I am sated, honest.

But some people exist in a mode of constant omnidirectional condescension, like little almighties, looking down in all directions.

Mostly lost causes. Deflating their egos sometimes helps, but usually just makes them worse.

so, I should provide some more context to that, I guess. my web server setup isn't "small" by most casual hosters definition. the total traffic usually is always above 5req/s, and this is not an issue for me.

also, crawlers are everywhere. not just those that I mentioned, but also search engine crawlers, and others. a big chunk of my traffic is actually from "bytespider", which is the LLM training bot from the TikTok company. It wasn't mentioned in this post, because although they make a lot of traffic (in terms of traffic size), that's primarily because they also ingest images, and their request volume is generally low.

some spiders are more aggressive than others. a long, long time ago - I've seen a crawler try to enumerate diaspora*s numeric post IDs to crawl everything, but cases like this are rare.

in this case, what made me angry was the fact that they were specifically crawling the edit history of the diaspora wiki. that's odd, because search engines don't care about old content generally. it was also odd, because the request volume was so high, it caused actual issues. MediaWiki isn't the best performance-wise, and especially the history pages are really, really slow. and if you have a crawler firing requests are multiple requests per second, this is bad - and noteworthy.

I've talked privately to others with affected web properties, and it indeed looks like some of those companies have "generic web crawlers", but also specific ones for certain types of software. MediaWiki is frequently affected, and so are phpBB/smf forums, apparently. those crawlers seem to be way more aggressive than their "generic web" counterparts - which might actually just be a bug, who knows.

a few people here, on mastodon, and on other places, have made "suggestions". I've ignored all of them, and I'll continue to ignore all of them. first, suggesting blocking user agent strings or IPs is not a feasible solution, which should be evident to everyone who read my initial post.

I'm also no a huge fan of the idea to feed them trash-content. while there are ways to make that work in a scalable and sustainable fashion, the majority of suggestions I got were along the lines of "use an LLM to generate trash content and feed it to them". this is, sorry for the phrase, quite stupid. I'm not going to engage in a pissing contest with LLM-companies about who can waste more electricity and effort. ultimately, all you do by feeding them trash content is to make stuff slightly more inconvenient - there are easy ways to detect that and to get around that.

for people who post stuff to the internet and who are concerned that their content will be used to train LLMs, I only have one suggestion: use platforms that allow you to distribute content non-publicly, and carefully pick who you share content with. I've gotten a lot of hate a few years ago for categorically rejecting a diaspora feature that would implement a "this post should be visible to every diaspora user, but not search engines" feature, and while that post was written before the age of shitty LLMs, the core remains true: if your stuff is publicly on the internet, there's little you can do. the best thing you can do is be politically engaged and push for clear legislative regulation.

for people who host their own stuff, I also can only offer generic advice. set up rate limits (although be careful, rate limits can easily hurt real users, which is why the wiki had super relaxed rate limits previously). and the biggest advice: don't host things. you'll always be exposed to some kind of abuse - if it's not LLM training bots, it's some chinese or russian botnet trying to DDoS or crawl for vulnerabilities, or some spammer network who want to post viagra ads on your services.

Maybe worth a try: Nepenthes

404media.co/email/7a39d947-4a4…

Developer Creates Infinite Maze That Traps AI Training Bots

"Nepenthes generates random links that always point back to itself - the crawler downloads those new links. Nepenthes happily just returns more and more lists of links pointing back to itself."

^{Jason Koebler (404 Media)}