I have around 10-20GB github / gitlab mirror. I am constantly under attack from crawlers from top US technology corporations and LLM startups. Whenever I ban one IP range they switch to other - I don't know if those fuckers have tickets in their systems to do it manually or they just deploy this shit all over the planet. From what I observe during attacks that I mitigate the best way to poison them is to just create gitea instance with poisoned code repository and couple hundred revisions. It's because what they are most interested in is html representation of diff between two git revisions.
this post was submitted on 13 Jan 2026
331 points (98.0% liked)
Technology
78786 readers
4664 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
Why isn't there anything in the DMCA for stopping crawlers? They have stuff about requiring crawlers to follow attribution and whatnot, but nothing for not allowing crawlers in the first place. Stupid as shit.
I can get a 50Gb/s residential link where I am, and have a whole rack of servers.
Sounds like a good opportunity to crowd fund thousands and thousands of common scrapeable instances that have random poisoning.
To be honest bandwidth isn't a problem because it's text files. The problem is to optimize network stack for multiple connections because they're hitting from whole subnets without any delay so literally ddos and cache those html files because at some point CPU becomes bottleneck.
This is assuming aggressively cached, yes.
Also "Just text files" is what every website is sans media. And you can still, EASILY get 10+ MB pages this way between HTML, CSS, JS, and JSON. Which are all text files.
A gitea repo page for example is 400-500KB transferred (1.5-2.5MB decompressed) of almost all text.
A file page is heavier, coming in around 800-1000KB (Additional JS and CSS)
If you have a repo with 150 files, and the scraper isn't caching assets (many don't) then you just served up 135MB of HTMl/CSS/JS alongside the actual repository assets.
I don't know from theory or counting but I know that my 8 cores depleted sooner than my bandwidth and I have like 60 Mb/s uplink. My linux network stack parameters are pretty aggressive. The way I figured out that something is not right was when I heard loud fan noise from my server inside room. I logged in and all cores were red and logs were showing corporate fuckers trying to burn my house.
I assume that the gitea instance itself was being hit directly, which would make sense. It has a whole rendering stack that has to reach out to a database, get data, render the actual webpage through a template....etc
It's a massive amount of work compared to serving up static files from say Nginx or Caddy. You can stick one of these in front of your servers, and cache http responses (to some degree anyways, that depends on gitea)
Benchmarks like this show what kind of throughput you can expect on say a 4 core VM just serving up cached files: https://blog.tjll.net/reverse-proxy-hot-dog-eating-contest-caddy-vs-nginx/#10-000-clients
90-400MB/s derived from the stats here on 4 cores. Enough to saturate a 3Gb/s connection. And caching intentionally polluted sites is crazy easy since you don't care if it's stale or not. Put a cloudflair cache on front of it and even easier.
You could dedicate an old Ryzen CPU (Say a 2700x) box to a proxy, and another RAM heavy device for the servers, and saturate 6Gb/s with thousands and thousands of various software instances that feed polluted data.
Hell, if someone made it a deployable utility..... Oof just have self hosters dedicate a VM to shitting on LLM crawlers, make it a party.
You won't get those numbers from internet requests, they do it locally or in cloud vpc, honestly those benchmarks are shit unless you are ISP. It's because you have ISP and your router involved before you even receive request. If you have traffic from all over the world there is also speed of light delay. Then you have linux tcp/ip stack and number of open files.
I use openresty, I could add lru cache on top but it doesn't even make sense because each bot just tries one unique request so you would have to generate html files manually instead of hosting gitea instance.
Gitea is on sqlite database on nvme so db doesn't really matter. I could put the sqlite on ramdisk as server is using UPS so I don't care about power outage but this would be ridiculous.
Anyways simplest way is just block ip ranges in firewall and move on.
Been thinking about making one of these too, especially since I have a catchy name : asbestos
Me too, but with procedural image generation. Use some templates which are put together with CPU blitter (extremely fast and effective), add some random descriptive text, then done. Don't know how much my theory would work IRL.
With the amount of AI generated horseshit out there already, they've already pissed in the well.
i would imagine companies would just filter it out
need some more clever way of hiding it or allow it to be self hosted so that it has various urls
So it would be effective at preventing your site from being used as training data.
If I am reading this correctly, anyone who wants to use this service can just configure their HTTP server to act as the man in the middle of the request, so that the crawler sees your URL but is retrieving poison fountain content from the poison fountain service.
If so, that means the crawlers wouldn't be able to filter by URL because the actual handler that responds to the HTTP request doesn't ever see the canonical URL of the poison fountain.
In other words, the handler is "self hosted" at its own URL while the stream itself comes from the same URL that the crawler never sees.
If, suppose, I were optimistic over this technology, but pessimistic over its current stage of development, I'd expect this to be a cure. It's a problem they'll have to solve. A test they'll have to pass.
If somewhere inside those things someone makes a mechanism building a graph of syllogisms, no kind of poisoned input data will be able to hurt them.
So - this is a good thing, but when people say it's a rebellion, it's not.
A test they’ll have to pass.
This makes me chuckle, as they invented euphemisms like 'hallucinations' because their LLM models can't do what they promise. Fabulous marketing, but clearly they didn't do enough testing.
load more comments
(3 replies)
Samsung and Anthropic published independently created data showing how little bad data it takes to effectively poison very large models. LLMs pretend to be complex, but they aren't, they'll not continue to improve at the initial rate we got used to seeing. Just ask OpenAI.
load more comments
(5 replies)
Not all problems may be cured immediately. Battles are rarely won with a single attack. A good thing is not the same as nothing.
"You're not opposing me. All you've done is create a problem that will stop me until I have it figured out." is the description of every struggle between opposing forces, so it's interesting that you disagree with that.
load more comments
(1 replies)
load more comments
(1 replies)
view more: next ›