Linux

8484 readers

327 users here now

A community for everything relating to the GNU/Linux operating system (except the memes!)

Also, check out:

Original icon base courtesy of lewing@isc.tamu.edu and The GIMP

founded 2 years ago

MODERATORS

Ategon@programming.dev

anzo@programming.dev

dwraf_of_ignorance@programming.dev

210

FOSS infrastructure is under attack by AI companies (thelibre.news)

submitted 4 months ago* (last edited 4 months ago) by throws_lemy@lemmy.nz to c/linux@programming.dev

38 comments fedilink hide all child comments

LLM scrapers are taking down FOSS projects' infrastructure, and it's getting worse.

you are viewing a single comment's thread
view the rest of the comments

[–] grrgyle@slrpnk.net 68 points 4 months ago* (last edited 4 months ago) (12 children)

Wow that was a frustrating read. I dd not know it was quite that bad. Just to highlight one quote

they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don’t give a single flying fuck about robots.txt, because why should they. [...] If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.

[–] jatone@lemmy.dbzer0.com 29 points 4 months ago (11 children)

the solution here is to require logins. thems the breaks unfortunately. it'll eventually pass as the novelty wears off.

[–] nao@sh.itjust.works 12 points 4 months ago (4 children)

Next you'll have to invest in preventing automated signups

[–] jatone@lemmy.dbzer0.com 1 points 4 months ago

not really, just tie it with 2fa SMS style and the hurdle is large enough most companies won't bother.

load more comments (3 replies)

load more comments (9 replies)