I just geo-restrict my server to my country, certain services I’ll run an ip-blacklist and only whitelist the known few networks.
Works okay I suppose, kills the need for a WAF, haven’t had any issues with it.
this post was submitted on 26 Nov 2025
124 points (97.0% liked)
Selfhosted
53155 readers
487 users here now
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.
Rules:
-
Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
-
No spam posting.
-
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
-
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
-
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
-
No trolling.
Resources:
- selfh.st Newsletter and index of selfhosted software and apps
- awesome-selfhosted software
- awesome-sysadmin resources
- Self-Hosted Podcast from Jupiter Broadcasting
Any issues on the community? Report it using the report flag.
Questions? DM the mods!
founded 2 years ago
MODERATORS
Fucking hell.
Yeah and that's why people are using cloudflare so much.
One corporation DDOS's your server to death so that you need the other corporations' protection.
That's a nice website you gots there, would be ashame if something weres to happen to it.
basically protection racket
A friend (works in IT, but asks me about server related things) of a friend (not in tech at all) has an incredibility low traffic niche forum. It was running really slow (on shared hosting) due to bots. The forum software counts unique visitors per 15 mins and it was about 15k/15 mins for over a week. I told him to add Cloudflare. It dropped to about 6k/15 mins. We excitemented turning Cloudflare off/on and it was pretty consistent. So then I put Anubis on a server I have and they pointed the domain to my server. Traffic drops to less than 10/15 mins. I've been experimenting with toggling on/off Anubis/Cloudflare for a couple months now with this forum. I have no idea how the bots haven't scrapped all of the content by now.
TLDR: in my single isolated test, Cloudflare blocks 60% of crawlers. Anubis blocks presumably all of them.
Also if anyone active on Lemmy runs a low traffic personal site and doesn't know how or can't run Anubis (eg shared hosting), I have plenty of excess resources I can run Anubis for you off one of my servers (in a data center) at no charge (probably should have some language about it not being perpetual, I have the right to terminate without cause for any reason and without notice, no SLA, etc). Be aware that it does mean HTTPS is terminated at my Anubis instance, so I could log/monitor your traffic if I wanted as well, so that's a risk you should be aware of.
You have to grow spikes and make it painful for bots to crawl your site. It sucks, and it costs a lot of extra bandwidth for a few months, but eventually they all blacklist your site and leave you alone.
AI scrapers are the new internet DDoS.
Might want to throw something Infront of your blog to ward them off like Anubis or a Tarpit.
the one with the quadrillion hits is this bad boy: https://www.babbar.tech/crawler
Babbar.tech is operating a crawler service named Barkrowler which fuels and update our graph representation of the world wide web. This database and all the metrics we compute with are used to provide a set of online marketing and referencing tools for the SEO community.
I run an ecommerce site and lately they've latched onto one very specific product with attempts to hammer its page and any of those branching from it for no readily identifiable reason, at the rate of several hundred times every second. I found out pretty quickly, because suddenly our view stats for that page in particular rocketed into the millions.
I had to insert a little script to IP ban these fuckers, which kicks in if I see a malformed user agent string or if you try to hit this page specifically more than 100 times. Through this I discovered that the requests are coming from hundreds of thousands of individual random IP addresses, many of which are located in Singapore, Brazil, and India, and mostly resolve down into those owned by local ISPs and cell phone carriers.
Of course they ignore your robots.txt as well. This smells like some kind of botnet thing to me.
I don’t really get those bots.
Like, there are bots that are trying to scrape product info, or prices, or scan for quantity fields. But why the hell do some of these bots behave the way they do?
Do you use Shopify by chance? With Shopify the bots could be scraping the product.json endpoint unless it’s disabled in your theme. Shopify just seems to show the updated at timestamp from the db in their headers+product data, so inventory quantity changes actually result in a timestamp change that can be used to estimate your sales.
There are companies that do that and sell sales numbers to competitors.
No idea why they have inventory info on their products table, it’s probably a performance optimization.
I haven’t really done much scraping work in a while, not since before these new stupid scrapers started proliferating.
Negative. Our solution is completely home grown. All artisinal-like, from scratch. I can't imagine I reveal anything anyone would care about much except product specs, and our inventory and pricing really doesn't change very frequently.
Even so, you think someone bothering to run a botnet to hound our site would distribute page loads across all of our products, right? Not just one. It's nonsensical.
Yeah, that’s the kind of weird shit I don’t understand. Someone on the other hand is paying for servers and a residential proxy to send that traffic too. Why?
It's 12181 hits and the number behind the plus sign are robots.txt hits. See the footnote at the bottom of your screenshot.
Phew, so I'm a dumbass and not reading it right. I wonder how they've managed to use 3MB per visit?
The robots are a problem, but luckily we're not into the hepamegaquintogilarillions... Yet.
12,000 visits, with 181 of those to the robots.txt file makes way, way more sense. The 'Not viewed traffic' adds up to 136,957 too - so I should have figured it out sooner.
I couldn't wrap my head around how large the number was and how many visits that would actually entail to reach that number in 25 days. Turns out that would be roughly 5.64 quinquinquagintillion visits per nanosecond. Call it a hunch, but I suspect my server might not handle that.
The traffic is really suspicious. Have you by any chance a health or heartbeat endpoint which provides continuous output? That would explain why so many hits cause so much traffic.
It's super weird for sure. I'm not sure how the bots have managed to use so much more bandwidth with only 30k more hits than regular traffic, I guess they probably don't rely on any caching and fetch each page from scratch?
Still going through my stats, but it doesn't look like I've gotten much traffic via any API endpoint (running WordPress). I had a few wallpapers available for download and it looks like for whatever reason the bots have latched onto those.
Check out Anubis. If you have a reverse proxy it is very easy to add, and for the bots stopped spamming after I added it to mine
I also recommend it.
Unknown Robot is your biggest fan.
does your blog have a blackhole in it somewhere you forgot about 😄