this post was submitted on 26 Nov 2025
321 points (96.8% liked)

Selfhosted

53179 readers
942 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

  1. Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don't duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago
MODERATORS
 

Got a warning for my blog going over 100GB in bandwidth this month... which sounded incredibly unusual. My blog is text and a couple images and I haven't posted anything to it in ages... like how would that even be possible?

Turns out it's possible when you have crawlers going apeshit on your server. Am I even reading this right? 12,181 with 181 zeros at the end for 'Unknown robot'? This is actually bonkers.

Edit: As Thunraz points out below, there's a footnote that reads "Numbers after + are successful hits on 'robots.txt' files" and not scientific notation.

Edit 2: After doing more digging, the culprit is a post where I shared a few wallpapers for download. The bots have been downloading these wallpapers over and over, using 100GB of bandwidth usage in the first 12 days of November. That's when my account was suspended for exceeding bandwidth (it's an artificial limit I put on there awhile back and forgot about...) that's also why the 'last visit' for all the bots is November 12th.

you are viewing a single comment's thread
view the rest of the comments
[–] dual_sport_dork@lemmy.world 2 points 23 hours ago

In my case the pattern appears to be some manner of DDoS botnet, probably not an AI scraper. The request origins are way too widespread and none of them resolve down to anything that's obviously datacenters or any sort of commercial enterprise. It seems to be a horde of devices in consumer IP ranges that have probably be compromised by some malware package or another, and whoever is controlling it directed it at our site for some reason. It's possible that some bad actor is using a similar malware/bot farm arrangement to scrape for AI training, but I'd doubt it. It doesn't fit the pattern from that sort of thing from what I've seen.

Anyway, my script's been playing automated whack-a-mole with their addresses and steadily filtering them all out, and I geoblocked the countries where the largest numbers of offenders were. ("This is a bad practice!" I hear the hue and cry from specific strains of bearded louts on the Internet. That says maybe, but I don't ship to Brazil or Singapore or India, so I don't particularly care. If someone insists on connecting through a VPN from one of those regions for some reason, that's their own lookout.)

They seem to have more or less run out of compromised devices to throw at our server, so now I only see one such request every few minutes rather than hundreds per second. I shudder to think how long my firewall's block list is by now.