Same thing used to happen with reddit giving sites the hug of death lol

It's unfortunate but kind of expected.

Sounds like every instance is doing the preview generation. Maybe it could grab the preview data it needs on the original instance, and pass that around with the post...

I added a rule in my .htaccess for their user agents to direct all requests to a static page on my domain.

Why not have the users fetch the previews themselves. The user's browser, that is.

Thanks for bringing this up. I think in general we need to work on optimisation of Activitypub and Threadiverse implementations of it.

So we need people to just mass post links to bad websites, is what you're saying.

it seems like ideally it shouldn't attempt to preview the link before a user tries to view the content. i guess that would increase latency for users though

Is this the same thing as the “Reddit hug of death”?

There are two proposed solutions I have seen. I do not claim that this is exhaustive, just only limited to the proposals that I have stumbled across.

FASP - Fediverse Auxiliary Service Provider

The idea that mastodon has is to have a third-party service that does something like generate a link preview once and then share that to fediverse instances that need it. They have a whole spec they wrote up about these kinds of services that can be run to help out fediverse instances with different things like this, search, or discoverability. They call these FASPs - the repo with the spec is here.

I don't yet know of any FASP that is live and running, so it is kind of a paper spec at the moment. I don't do too much on the microblogging side of the fediverse though, so I very well could be wrong.

Federate link metadata and the post at the same time

This comes from an issue on PieFed's repo. It is basically just an idea at this point, but it is a pretty basic one; have the originating instance fetch the link preview information and simply federate it out alongside the post itself.

Is there a reason why the publishing instance is not downloading the preview image, and then submits it with the post to the other instances? This way they would not have to fetch it themselves.

I have my blog address (which goes to my home server) in my Mastodon profile, and every time I post on Masto, my server gets a request from every Mastodon site that federates with my instance - i.e. hundreds of requests. It's no problem for my server, because my blog is a static site, but for anything that dynamically generates every single page impression (like worpress or joomla), it could be a problem. Caching settings might help, I suppose.

Truth is, though, I get far more traffic from vulnerability scanners and AI crawlers that don't honour robots.txt (or aren't yet listed).

It's almost like the software that powers the internet was designed for a different time and is only still used because there's monied interest in keeping it around. We need GNUnet.

Lemmy does this to itself, too - when you create a post it federates out and then every Lemmy instance that receives it makes a GET request to retrieve the post from the original server. Even though it just received the exact same data. Some other fedi software is even worse - Akkoma does a GET of the id of every activity it receives, not just for posts but for Likes too.

It's insane.

Anyway, proper caching at the reverse proxy / Cloudflare level solves it. The same kind of caching setup would do the same for sites serving HTML, like NPR too.

I mean, just how many instances are gonna ping the URL? Several hundred, low thousands at worst?

Unless it happens to be some very large media content or a dynamic page with no caching, I'd think the automated federation hits shouldn't be too bad.

The influx of users that might come after a viral post though, that's rough....

Heh. Ever heard of the Slashdot effect?