sunaurus

joined 2 years ago
[–] sunaurus@lemm.ee 5 points 2 weeks ago (2 children)

We'll get some more control over images when 1.0 comes, I will take a look at it again then, but in general the new user cooldown for image uploads has really significantly reduced the amount of image related abuse we have seen.

@Character_Locked@lemm.ee please understand, nobody on our admin team is getting paid to moderate disgusting images, they are volunteering to do it, and it takes a mental toll. Also, image hosting itself is not free. Our donations right now are barely breaking even with server costs. So realistically, our options are not "current situation vs fully unrestricted image uploads", they are "current situation vs completely burned out admin team and costs increasing beyond donations".

391
submitted 1 month ago* (last edited 1 month ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hey folks!

I'm writing this because funding for the Lemmy project has dropped to critical levels, which could seriously impact its future development.

Thanks to the generous support of our lemm.ee community, our server infrastructure costs are covered, and we even have a few months of runway. I'm deeply grateful to everyone who has contributed - lemm.ee wouldn't exist without your help.

However, infrastructure alone isn’t enough. Our servers run Lemmy software, and without ongoing development, the platform cannot grow or even be maintained.

Lemmy is an open-source project with many contributors, but the vast majority of development work has been carried out by a small group of core maintainers. A few maintainers work full-time on the project, relying solely on donations and occasional grants to support themselves.

I've seen Lemmy development up close, and the maintainers have consistently gone above and beyond what I consider the standard for small open-source teams - they are constantly writing code, mentoring contributors, and keeping everything running. Their work is essential, and without continued support, it cannot be sustained.

If you value Lemmy, please consider supporting its maintainers directly. Every bit helps.

Please check out this post for more details about how to support the maintainers: https://lemm.ee/post/63034576

Thank you for reading, I hope you have a great weekend!

[–] sunaurus@lemm.ee 3 points 1 month ago (2 children)

Our servers are all hosted in Finland indeed, but we use Cloudflare for DDoS protection etc. Clients/bots/scripts/etc (like FediDB) don't see our server IP addresses directly, they see Cloudflare IP addresses. FediDB uses the IP address it sees to assume what country the server is in.

The same is true for a bunch of other Lemmy servers as well - a lot of the ones FediDB shows as "United States" are actually hosted in Europe (or elsewhere).

[–] sunaurus@lemm.ee 2 points 1 month ago (1 children)

Most applications are handled automatically, a smaller percentage is sent to manual approval for admins. This is based on different signals and the aim is to potentially reduce some abuse, or at least make it a bit harder.

Median time for accepting applications in the past month on lemm.ee was 1 minute and 42 seconds. And that includes the time it took for users to manually verify their e-mails first.

[–] sunaurus@lemm.ee 0 points 2 months ago (1 children)

What I mean is that some posters have a bunch of alts on different instances, and if they get awards split between their accounts, then they'll be at a bit of a disadvantage 😄

[–] sunaurus@lemm.ee 0 points 2 months ago (3 children)

This is cool!

Any rules for alts? Can people "claim" alt accounts to sum up their votes?

[–] sunaurus@lemm.ee 0 points 2 months ago (1 children)

It's very unlikely lemm.ee will ever be a suitable host for video content. Please host your videos on other platforms, embedding them on lemm.ee posts will work just fine.

[–] sunaurus@lemm.ee 6 points 2 months ago (1 children)

Whenever you give your free time and energy to build something open source for the world, there are always some people who say “why aren’t you giving me 10x more, you lazy dev” 😅

2
submitted 2 months ago* (last edited 2 months ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hey folks!

For the past few hours, lemm.ee has been bombarded with abnormal (almost definitely automated) traffic from a range of different IP addresses. This managed to overwhelm our servers, and we were offline for the past hour or so.

I was in the middle of celebrating my birthday, so response was a bit slow, but I believe we are recovering now, with mitigations in place to try and prevent further issues. Some of you may be inconvenienced by some bot checks when you browse lemm.ee, I am sorry about that, but it's necessary for now.

Sorry for the issues and I hope you have a nice weekend ahead!

4
submitted 3 months ago* (last edited 3 months ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hi folks!

Over the past few months, we have started seeing a significant amount of new user sign-ups. I would like to take this opportunity to welcome all of our new members, and to share some useful resources and info about lemm.ee.

First, some stats

Here is a bar chart of daily new users (this is only counting users which have been approved by our admins):

As you can see from the chart, for most of 2024, we were accepting roughly around 10-20 new users every day. Then, from the start of this year, the daily numbers have been constantly growing. Yesterday, we approved a massive 609 new users on lemm.ee.

The increase in sign-ups is significant enough that I have been taking several steps to improve our monitoring & anti-bot measures, but so far, it seems the vast majority of the new users are completely legitimate real humans! (Thank you all for not being bots 😅)

About lemm.ee

This Lemmy instance is turning 2 years old very soon. It was initially created around the time of the Reddit API changes, when existing Lemmy servers were getting overloaded with new users - lemm.ee was intended to help spread the load. We're now the second largest Lemmy server when it comes to monthly active users.

Our core philosophy for this instance has always been to treat it as a generic gateway to the Lemmy network. I want to provide our users a stable and reliable home for their Lemmy account, so that they can have easy access to all of their communities, regardless of what instance the community is actually hosted on.

We run on some decently beefy hardware, and our setup is fairly customized in several ways in order to ensure a smooth experience for our users (most of the time, this has worked out quite well!). Our servers are currently hosted in Finland.

Our infrastructure has been funded by the community almost from the start through GitHub sponsorships and Ko-Fi donations. I am sure I speak on behalf all of our users when I say that I am extremely grateful to all supporters - you are really responsible for the continued existence of this instance!

Lemmy itself is open source software, and while it has improved massively during the time I have been using it, it definitely still has some rough edges. Please be patient when using Lemmy, and remember that it is being built collaboratively by humans (not corporations), without any intent of ever turning it into a business.

Useful resources

Don't forget to participate!

Communities on Lemmy only work if people actively use them. Even upvoting/downvoting based on quality of content is a great start, but I would really like to encourage you all to comment and even write posts, because that's really the best way to build communities.

If you have any questions or thoughts about lemm.ee or Lemmy in general, feel free to post a comment below this post, and myself or one of our veteran users will definitely respond.

I hope you enjoy your time on lemm.ee, and I wish you all a great week!

 

Hey folks

Just a quick heads up, we will be performing some database maintenance today. Expected downtime is ~15 minutes.

Sorry for the inconvenience!


Update: maintenance complete!

1
submitted 5 months ago* (last edited 5 months ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hey Folks

Just a quick note to let you all know about some changes in the lemm.ee admin team. After discussing things with the other admins, we've decided to shuffle around our roles a bit.

Up until now, I’ve been the head admin at lemm.ee - handling infrastructure, maintaining rules and policies, and acting as the main contact person for the admin team.

However, I’ve come to realize that this role has taken a toll on me. While I still love the idea of Lemmy and everything it stands for, being an admin has slowly drained the joy I once had for the platform. The occasional negative experiences have been increasingly difficult for me to shake off. For the past several months, I’ve found myself hesitating to check my DMs or the moderation queue, simply because I’m bracing for some new drama that I no longer have the energy to manage.

After some conversations with the team, we’ve agreed on a plan to ensure my burnout doesn’t negatively impact the instance:
  1. I am stepping down as head admin of lemm.ee.
  2. The new main contact person for the admin team will be @EllaSpiggins@lemm.ee.
  3. I’ll continue to maintain and update the infrastructure behind the scenes.
  4. The rest of the admin team will now handle all moderation issues, managing our policies, and any general admin communications.

It’s been an honor to serve as your head admin, and I’m incredibly grateful for the amazing people I’ve met here. I’m excited to stay involved in a capacity that works better for me and allows me to enjoy this community once again.

See you around!

 

Hey folks!

I'll be updating lemm.ee to 0.19.7 shortly. I'll take it offline & also perform some database maintenance at the same time.

I have been investigating some incoming federation issues, and I haven't been able to find the actual cause. My current theory is that they are related to some server timeouts. I am hoping that the combination of the Lemmy upgrade and database maintenance may help improve the situation, but if not, then I will try other approaches.

Edit: the upgrade has been completed!

 

Hey!

Unfortunately, Hetzner (our hosting provider) is currently experiencing some network issues. They are planning to address this with an emergency maintenance in roughly 13 hours from now, which will cause lemm.ee downtime. Hopefully we'll be fully recovered later tomorrow!


UPDATE: Sorry for the false alarm, I was on the move when I posted this and missed the fact that the Hetzner notice was actually for next month! So it's not as imminent as I originally understood. As we have a whole month to prepare, I will probably be able to come up with some alternative solution to prevent the downtime while they are conducting this maintenance.

 

Hey folks!

I am looking for feedback from active lemm.ee users on what you all value when it comes to images on Lemmy. I'll go into a bit of detail about what our options are, and then I would ask you to voice your opinion about the issue in the comments.

First, some context for those who don't know. Lemmy software can be configured to handle images in three different ways:

  1. Store images locally - whenever an external image is posted somewhere, lemm.ee will download a permanent local copy. When you view posts, you are seeing our local copy of the image.
  2. Proxy all images - similarly to the first option, lemm.ee will download a local copy of external images, however, this copy is temporary. It will be automatically deleted shortly after, and if users open the relevant post/comment again in the future, there will be another attempt to download a temporary copy at that point.
  3. Pass through external images directly - lemm.ee never downloads any external images, users will always connect directly to the source servers to load the images.

There are pros and cons to each configuration.

Storing images locally

Benefits:

  1. Your IP address is never leaked to external image hosts, as you never connect directly to the source server. External image hosts only see the IP address of the lemm.ee server.
  2. External servers don't become bottlenecks for opening lemm.ee posts. If an external server is slow, it won't matter, because the image is always available locally

Downsides:

  1. As time goes on, our storage will fill up with hundreds of gigabytes of useless images, most of which will never be viewed again after the relevant posts fall off the front page.
  2. Many big external image hosts will rate limit bigger Lemmy servers, causing broken images when we fail to make a local copy.
  3. Crucially: some people love to spend their time uploading illegal content to online servers. There are tools to try and filter out such content, but these are not perfect. The end result is that there is a high chance of some content like this inadvertently reaching lemm.ee storage and staying there permanently. This downside is why lemm.ee has not, and will not, use this particular configuration.

Proxying images

Benefits: In addition to the same benefits as exist for the permanent local storage, by only temporarily making local copies for the moment they are requested by our users, we free up a ton of storage & remove the risk of permanently storing illegal content on our servers.

Downsides: The key downside is that external rate limits hit us much harder, as we will be requesting external images far more often. This results in a lot of constant broken images on lemm.ee.

Passing through external images

Benefits:

  1. Images are rarely broken, unless the source server goes down.
  2. The images never touch our servers, removing a lot of risk with illegal content as well as with storage costs.

Downsides:

  1. Our users lose a degree of privacy. Every external image that is loaded on your browser will result in the remote server getting a request directly from your computer to fetch that image - this is pretty much the same as you had visited that external server directly, which lets them log your IP address if they wish.
  2. When remote servers are slow, it can slow down the entire page load in some cases.

Current situation

Initially, lemm.ee was using the third option of passing through images. Ever since support for option 2, image proxying, was implemented in Lemmy code, we immediately switched to that option, mainly for the privacy benefits. However, after many months, and being blocked by more and more external servers, it is clear that image proxying is seriously degrading the user experience on lemm.ee. We often end up with broken images, and our users have to deal with the results.

I still believe image proxying is a really valuable feature, but I am starting to believe it is a better fit for small instances which make much less requests to external servers.

As a result, I am now seriously considering switching back to the previous method of passing through external images.

This is where you come in - I would ask you as users to please let me know which do you value more: the privacy that you get from image proxying, or the better user experience you get from directly passing through images from their source. Please let me know in the comments how you feel. If I get enough feedback about people being against image proxying, then I will be switching it off for lemm.ee soon. Thanks for reading & sharing your thoughs, and I hope you have a great weekend!

12
submitted 11 months ago* (last edited 11 months ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hey folks!

For anybody stumbling on this post from outside lemm.ee: I am the head admin of lemm.ee, a general purpose Lemmy instance, which recently turned 1 year old. I am writing this post to elaborate on how we approach defederation on lemm.ee.

Anybody who has been on Lemmy for a while has most likely seen several public defederation drama posts (most recently regarding lemmy.ml, but there have been many many others previously). As an admin, I have probably seen far more than what is visible publicly, as I regularly receive private messages on the topic, ranging from polite questions about federation, to outright demands that I immediately defederate, and even to threats and personal attacks over the fact that I have not defederated some particular instance. It is definitely a topic that will keep coming up for as long as Lemmy exists, which is why I feel it would be useful to condense my current thoughts about it in a single place.

Note that while I strongly believe everything this post contains, it is definitely a subjective topic, and there is no single right answer here. Other instances have completely different approaches to federation compared to lemm.ee, and that’s of course totally fine. The beauty of Lemmy is that everybody can choose their home instance, and in fact, everybody is free to spin up their own instance and run it however they feel is best. For an absurd example, if you want to create an instance which defederates any instance with an “L” in their name, then nobody can stop you!

Quick intro to the lemm.ee federation policy

Very shortly after creating lemm.ee, I wrote down a federation policy, which basically boils down to “we treat defederation as an absolute last resort, and we do not use it as a generic way to curate content for lemm.ee users”. This policy can always be found in the sidebar of the lemm.ee front page.

In practice, this has meant that we have had extremely few defederations, and that we mostly solve problems with other means. I am very happy with the results, as it means that lemm.ee has become a great entry point into the Lemmy network, with very few artifical limitations on who our users are allowed to interact with.

The benefits of federation

I hope that this part of the post is very uncontroversial, but I firmly believe that federation is the absolute strongest feature of Lemmy. While we all know that the concept of federation can cause confusion for new users, this is usually overcome extremely quickly (for example, using the common e-mail providers analogy to explain Lemmy instances). To me, it’s completely clear that the benefits of federation far outweigh the downsides.

For example, by splitting the Lemmy network between thousands of independent nodes, we ensure that:

  1. Any single entity is not a single point of failure for the whole network. Even if the biggest instance goes down tomorrow, their content will still be accessible through all the other federated instances.
  2. The maximum impact of admins is limited to their own instance. As a lemm.ee admin, I can ban a remote user from posting on lemm.ee, but I can’t completely ban them from the entire network.
  3. Private user data (such as ip addresses, e-mails, etc) are never shared between instances. No single malicious instance can harvest user data for the entire network, and extremely privacy sensitive users can always spin up their own instance if they don’t want to put their trust in any existing admins.

One thing which is probably important to note here is that I tend to view Lemmy instances as infrastructure, rather than as communities. I know that there are alternative approaches, as quite a few large instances are in fact run as mega-communities, but that’s not the approach I take with lemm.ee, because I feel like such an approach encourages centralization and negates some of the benefits of federation (if all communities related to one topic condense on a single instance, then that instance does effectively become a single point of failure for a large number of users).

In general, I feel like it should be a goal to encourage and cultivate decentralizing the network through federation as much as is practical, in order to maximize the above benefits.

The downsides of dedeferation

Conversely, defederation has a lot of downsides.

  1. It obviously negates all the benefits of federation mentioned above. Every time two instances defederate, the Lemmy network becomes less redundant, some communities become a bit more centralized, and the danger of malicious admins for those communities becomes much greater.
  2. There is a lot of collateral damage. The most common reason I have personally seen for defederation demands is related to moderation of either a single user, or a handful of users. For example, a lemm.ee user gets into some heated arguments with people from an instance with hundreds of active users, and then links this heated thread to me as proof that the instance should be immediately defederated. However, in this situation, there are hundreds of other users who were not even involved (or even aware of) the thread in question. By defederating, I would be making a decision to cut off every single lemm.ee user from every single one of those hundreds of innocent remote users.
  3. Ironically, defederation actually makes moderation more difficult. It was recently pointed out to me by a user on another instance that they are afraid they can’t effectively moderate communities on lemm.ee, because their instance has defederated several other instances, which means they would not be able to see posts from those instances on lemm.ee communities.
  4. It is extremely easy for malicious actors to abuse. In the year I’ve been on Lemmy, I have already seen two separate cases of users creating accounts on another instance and posting garbage, and then going back to their home instance and demanding their admins defederate over the content they themselves created. Basically, if an instance is known to use defederation as a tool to punish misbehaving users on other instances, then it’s actually quite easy for users to manipulate the situation to a place where admins have no alternative except to defederate.

It seems to me that a lot of users don’t think of such downsides when demanding defederation, or they just don’t consider them as important enough. In my opinion, these are all significant issues. I do not want to end up in a fragmented Lemmy network, where users are required to have accounts on 5 different instances in order to be able to access all their communities.

What’s the alternative to defederation? Should Lemmy become some kind of unmoderated free speech abolutism platform?

I want to be very clear that I do NOT believe in unmoderated social networks. Communities should always be free to set and enforce rules which foster healthy discussions. On top of that, instances should always be free to set and enforce rules for all of their users and communities.

In the case of lemm.ee, we have some instance-wide rules, and we will enforce them on all lemm.ee users, as well as all remote users participating in communities hosted on lemm.ee. For example, we never want to offer a platform for bigotry, so we regularly issue permanent bans for users who want to abuse lemm.ee to spread such hate. In practice, site bans have been extremely effective at getting rid of awful users, whether they are remote or local.

On top of site bans, Lemmy admins also have the option of removing entire remote communities. There are certainly cases where a community might be allowed on instance A, but not instance B - rather than defederating (and potentially cutting off a lot of innocent unrelated users), instance A can just “defederate” a single community.

Finally, a lot of issues can be solved through simple communication between instance admins. Often having a discussion with another admin results in pretty clear alignment over whether some user is problematic, and the user will end up being banned on their home instance.

Being one of the most openly federated large instances with such an approach, we have discovered several things:

  1. If we were to defederate over every rule breaking user or community on the Lemmy network, we would not be federated with any of the large instances at this point
  2. In the vast majority of cases, remote users who have broken lemm.ee rules have ended up banned on their home instance anyway - there is very little additional moderation workload for our admins from being widely federated
  3. If a user truly wants to spread some kind of hate, defederation wouldn’t stop them anyway, as they will just create accounts on any instance which they want to “attack”

The longer I run lemm.ee, the more sure I become that in the vast majority of cases of abusive users, the best approach is to simply hand out site bans.

When is defederation the only option?

Having said all of the above, I still believe that there a few cases when defederation is the best option:

  1. When an instance is abusing the Lemmy network - generating spam, advertising, illegal content, etc - either deliberately, or through inactive admins (this has been the most common reason for lemm.ee to defederate any instance in the past)
  2. When an instance is just causing too much moderation workload. So far, we haven’t experienced this yet on lemm.ee, but I can’t rule out that it could happen in the future.

Conclusion

I hope this post helps clarify my stance on defederation. Like I said in the beginning, I realize a lot of this is subjective, and there are no right or wrong answers - this is just the way we have been (and will be) doing things on lemm.ee. I intend to save this post and link it in the future when people bring up defederation requests. If you feel like I didn’t address something important, please feel free to raise it in the comments!

[–] sunaurus@lemm.ee 0 points 1 year ago

Basically, yes!

For the backend: our traffic is load balanced between multiple servers, so I can just spin up a new server with the latest version of Lemmy, add it to the load balancer, and then start taking down the servers with older versions. That way, there is no disruption at all for users, because there is always a server available to handle incoming traffic. The only requirement for this is that the new version of Lemmy can't have database migrations which break the old running servers.

For lemmy-ui: it's a bit more complicated, because even with a load balancer, it's still possible that one user making multiple sequential requests can end up getting responses from different servers. This is problematic, because during an upgrade, files from the new version are not available on the old servers, and vice versa. Fortunately, there are many ways to solve this problem, for lemm.ee, the solution I use is to just always serve lemmy-ui files from object storage, for all versions. In other words, after I upload lemmy-ui files for a new version, these will immediately also be available on old servers.

[–] sunaurus@lemm.ee 0 points 1 year ago (2 children)

By the way, as a mini-present, I have sneakily updated our Lemmy to 0.19.4! It was possible to do this one without any downtime, so I just did it quietly in the background.

0
submitted 1 year ago* (last edited 1 year ago) by sunaurus@lemm.ee to c/meta@lemm.ee
 

Hey, folks!

Today, we can celebrate the first anniversary of the creation of lemm.ee! I thought it would be cool to write down how lemm.ee was born, as well as collect some stats about our first year. Here goes!

A quick recap of the beginning of this instance

As probably many others here, I discovered Lemmy early last summer. I had been aware of the Fediverse previously, and always thought it was an amazing concept, but I had never been super interested in Twitter-style social networks. When I found out that Lemmy combined all the great parts of federation with the best parts of link aggregation, I knew that I had to join immediately.

As I was trying to find an instance to make my account on, I realized that most instances were struggling to keep up with a massive influx of new users. At the time, there was a big explosion in Lemmy user numbers, and the network wasn’t fully ready for it. I have some experience with building software for scale, so it felt natural to set up a new instance and try to help with spreading out the load. I got to work in the evening of the 8th of June, 2023, and I was actually so excited about everything, that I completely skipped sleep that night. By the morning of the 9th of June, lemm.ee was online.

From the very beginning, I always intended for lemm.ee to be a welcoming, reliable, and stable gateway into the Lemmy network. I wrote a welcome post on lemm.ee, which most of you have probably seen, as well as a comment on lemmy.ml inviting new users to lemm.ee (lemmy.ml, as many instances, was extremely overloaded at the time).

We started growing extremely quickly. Thousands of users joined lemm.ee over the first few months. Even during the biggest waves of new users, we never closed our sign-ups. The first month or two were definitely very stressful in terms of just trying to deal with the load, but overall, I think I managed to deal with it well enough, and lemm.ee has been running more or less smoothly (with a few exceptions) ever since.

Some stats about the first year

I promised to collect some statistics about lemm.ee so far. This is what I’ve come up with:

Usage

Overall, lemm.ee has 28,715 registered users. Of course, it’s easy to create an account, and most of these are probably inactive at this point, but it’s still a ridiculous amount.

Of all the registered users, 7903 have made at least one post or comment.

7373 users have never made any posts or comments, but have still been voting. This means that out of users who actually interact on lemm.ee, more than half generate content (through their comments and posts) - this is way more than I expected!

Meanwhile, we also have 13,439 users who have never made a single comment, post or vote. I guess most of these are people who just signed up and never got into Lemmy, but I’m sure there are quite a few hardcore lurkers among this group as well.

As for communities, our users have created 1430 of them. Most of these have not (yet) taken off, as only 491 of these communities have at least one comment in them. In general I am happy to see some great communities appearing on lemm.ee - my hope is that we can spread awesome communities out quite evenly on the network, so that in the end, no instance becomes a single point of failure for Lemmy.

Judging by posts and comments made by lemm.ee users, I feel like we’re definitely on the right track: our users have made 20,898 posts in local communities, and 30,847 posts in communities hosted on other instances. The situation is even better for comments, where lemm.ee users have written 42,785 comments in local communities, and a whopping 569,730 comments on remote communities! This means that lemm.ee is not just its own little closed pocket in the Fediverse, but indeed a proper gateway to the Lemmy network, which is exactly what I always hoped it would be.

Note about comment and post counts: I realize the numbers above don’t match the stats about posts and comments on our front page, I’m guessing something is out of sync there, but the stats I am sharing here are based on actual fresh data, counted directly in our database today.

Lemmy (and lemm.ee) would be quite useless without its users, so a big thanks to all of you for using lemm.ee!

Administration

We have a really awesome volunteer admin team, with admins putting in countless hours of their free time to help weed out bad actors. A lot of the work our admins do is completely invisible to most users. I think the admin team does not really get enough recognition, and in fact in many cases, they actually get some undeserved abuse thrown at them.

I am personally very grateful for everybody who has stepped up to be a part of the team, and I think all lemm.ee users benefit from their work every day. In the past year, our admins have handled 12,329 reports from users. While most reports aren’t too bad, and don’t require harsh action, there is still a significant amount of these reports which contain the absolute worst content which you can find on Lemmy - hate speech, bigotry, gore, even illegal content. Our admins are constantly going through every single report they receive, to ensure that mods are getting admin-level support where needed, and to ensure that malicious users in general can’t use lemm.ee to spread garbage into the Lemmy network.

In terms of admin actions, I think the most interesting statistic might be amount of users banned by lemm.ee admins, grouped by their home instance. I will list the top 10 instances here:

  • kbin.social: 581
  • lemm.ee: 355
  • lemmy.world: 31
  • sh.itjust.works: 29
  • m.mxin.moe: 28
  • discuss.tchncs.de: 26
  • kbin.chat: 22
  • mastodon.social: 19
  • lemmy.ca: 18
  • fedia.io: 16

As you can see, with the exception of kbin.social, the vast majority of our instance bans are for our own users. Most of the big instances are actually very good at banning their own abusive users, and once they are banned on their own instance, our admins don’t really need to worry about them, as they have no way to log in at that point. kbin.social is a bit of a special case - they either don’t give out a lot of bans, or those bans just don’t federate to Lemmy properly, and for some reason, a lot of advertisers sign up on that instance all the time.

Financials

I have received some questions every now and then about how much it costs to run lemm.ee. While you can always get a sense for the predicted monthly costs for the current month on https://status.lemm.ee/, I thought I might include a full breakdown of our costs for the first year here.

Here are all of our costs for the past year, grouped by service:

  • Postmark: 177.06€
  • Cloudflare: 222.28€
  • DigitalOcean: 1744.27€
  • Hetzner: 510.20€ (lemm.ee migrated from DigitalOcean to Hetzner several months ago)
  • Backblaze: 3.78€ (we’ve been using Backblaze B2 for a few months now, it’s incredibly cheap)
  • Domain registration: 100.70€ (paid for the next 10 years!)

We are currently completely funded by lemm.ee users!

There is a small minority of users who are shouldering the entire cost of lemm.ee for all of us. I am extremely grateful that others find Lemmy useful enough that they have put their own money into ensuring financial stability for lemm.ee.

We currently have 49 active sponsors on GitHub, and 7 active supporters on Ko-Fi. In addition, there have been 62 more sponsors on GitHub over the past year, as well as 49 additional supporters on Ko-Fi. This means that in total, 167 users have supported lemm.ee financially. This has completely exceeded all my expectations, I really think it’s incredible. A huge thanks on behalf of myself (and I think I can speak for all other lemm.ee users here as well) to all the supporters!

Conclusion

Running lemm.ee has certainly been a rollercoaster in many ways. There are a lot more things which happened during the first year that I could write about here. On the other hand, this post is already quite long, and a lot of the things which happened are probably best forgotten about anyway, so I think I should wrap up here 😅.

At its core, Lemmy is really an amazing piece of software. It’s helping real humans connect on the internet, without any corporate bullshit. I am very happy to be here with all of you, thank you for joining lemm.ee for its first year, and I hope you’ll join me here again when I write this post in another year from now!

 

Hello!

I am sunaurus, the head admin of lemm.ee. Ever since I created my instance, I have been following a lot of public and private discussion channels between different parties involved with Lemmy. As I’m sure many others have also noticed, the discussions in such channels sometimes get heated, and in fact recently, I feel like there has been a constant trend in these discussions towards a lot of demands, hostility, negativity, and a general lack of empathy between different participants in the Lemmy network.

I am writing this post for a few reasons:

  1. I would like add a bit of positivity by expressing my gratitude towards every single person who has helped improve Lemmy.
  2. I want to speak up in defense of different people who have been receiving negativity lately.
  3. There are a few false rumors spreading on Lemmy, which I would like to try and counteract with very simple evidence.
  4. I want to remind everybody that at the end of the day, all of us care about building and improving Lemmy. We all have the same goal, and it’s too easy to lose sight of that.

I will split up what I want to say in this post by different user groups - users, mods, admins and developers. I understand that many people belong to several (or even all) of these groups, but I just want to highlight the value of, and express my gratitude to each group separately.

Users

At the end of the day, Lemmy would not be worth anything without the users. Users bring Lemmy to life by posting great content, getting involved in discussions in comments, helping surface interesting content for others through voting and even keeping the platform clean through reports. I am extremely thankful for all the users who have given me so much enjoyment on this platform.

I believe that users often get treated unfairly on Lemmy based on what instance they are participating from. I’m sure so many of you have noticed comments around Lemmy along the lines of “Oh, another user from , I’m going to completely ignore your stupid takes”. I’ve also many cases of people treating users as second-class citizen if they are not on the same instance - for example, I’ve seen users who are active and valuable participants in communities on another instance receive comments like “why are you participating in our discussions, go back to your own instance”. In my opinion this is completely counterproductive to the whole idea of federation. On a human level, I can understand it - you’re far more likely to notice or remember what instance somebody is posting from if you have a negative experience. As a result, as time goes by, people tend to develop negative views of each instance, despite potentially having had many positive interactions with other users of those same instances. The message I want to put out here is that instances, especially bigger ones, are not monoliths - do not judge users based on what instance they are browsing Lemmy from, judge them by their actual words and actions.

Mods

There are some excellent communities already on Lemmy, and these communities are all continuously being built up and maintained by mods. Mods put in huge amounts of their free time and energy in order to provide spaces for all Lemmy users. They form the first line of defense against bad actors, they keep communities alive and often receive no praise, only criticism. I am very grateful to everybody who has dedicated time to building communities on Lemmy.

Users rarely notice the lengths mods go to in order to keep communities running smoothly - mods more often than not only get noticed when users disagree with some mod actions. I believe mods deserve a lot better than this. Constructive criticism can of course be useful to improve communities, but it must be balanced with empathy and kindness towards people who have been putting in effort to provide something for users. Remember that there is another human being reading your words when you start writing about the mods of any particular community. Users who are not happy with mods of a certain community always have the opportunity to start their own community and run it as they like.

Admins

Admins provide two main key functions for the network:

  1. Taking care of the actual infrastructure of Lemmy
  2. Working as a higher level defense against bad actors, in cases where mods are not enough

I can tell from my own experience that being an admin of a bigger instance requires constant energy and attention. I don’t believe that there is a single medium-to-big instance where the admins have not put in hundreds (if not thousands) of hours of their free time, as well as in many cases, probably their own money. This is a service which admins provide for free, and it is necessary in order to keep the Lemmy network healthy. I have endless respect for anybody who is willing to put themselves in the position of a Lemmy admin.

I have seen awful messages towards admins from all the other groups listed here, including other admins. These messages range from condescending and rude, to downright hateful. I have seen admins treated as useless and their work taken for granted. I have seen people getting frustrated with admins for not spending every waking minute on Lemmy. I have seen some users consistently spreading provably false rumors about particular admins in an effort to tarnish their reputation on Lemmy.

Before you take out frustration on admins, please remember that they are also humans who have been working tirelessly to improve Lemmy in their own way.

Also, a reminder: the absolute best feature of Lemmy is that users are free to pick their instance - and as a result, users are also free to pick their admins. Even more than that, users can always become their own admins by spinning up their own instance. Yes, this requires dedication, effort, and research, but that’s exactly my point. It’s not easy running an instance, and mistreating people who do this as a free service is completely unacceptable.

Developers

Lemmy development has been lead by a few key maintainers, with a massive amount of smaller contributors. The software is constantly being improved at a very good pace, and everybody is able to benefit from this effort at no cost whatsoever. I am extremely grateful to everybody who has participated in the development of the Lemmy software, and other related software, as without you folks, none of us would even be here now.

There seems to be a huge amount of people with very little appreciation of the work that has gone into the software. I’m sure many of you have seen countless messages where people express that the devs should be doing more in one way or another. “They should work faster”, “they should prioritize this obviously most important feature”, “they should be available 24/7 to offer support”, etc. I just want to take a moment here and acknowledge what core maintainers have already done for Lemmy:

  • Years worth of work on the code itself
  • Offering support to the community and other admins
  • Reviewing literally thousands of pull requests on GitHub
  • Acting fast in stressful situations where the Lemmy network has been overloaded
  • Not abandoning the project in the face of constant hateful users
  • Sacrificing literally hundreds of thousands of euros in missed salaries which they could have been getting if they were working for a tech company instead of working on Lemmy

I also want to take this moment to discredit some rumors which I have seen repeated too many times:

  1. Rumor: Lemmy devs do not accept outside code contributions

This is completely false - the maintainers are completely open to (and even constantly asking for) contributions. When somebody starts contributing, they will receive support and code reviews very quickly. I can tell you that I have experienced this myself several times, but that’s anecdotal, so let me also provide evidence:

a. Contributors list for the Lemmy backend: https://github.com/LemmyNet/lemmy/graphs/contributors

b. Contributors list for Lemmy UI: https://github.com/LemmyNet/lemmy-ui/graphs/contributors

Both of these lists include 100 different names, and that’s only because GitHub literally caps these pages to 100 users. Actually, the amount of different contributors is even bigger. If Lemmy devs did not accept and encourage outside contributions, then there would be no way for these lists to be so big.

  1. Rumor: Lemmy devs work too slowly

This is an extremely entitled and frankly stupid claim. I try to keep on top of the changes made in the Lemmy repo, and let me tell you, the pace of improvement is very good.

I very firmly believe that if the network started downgrading to Lemmy versions from ~8 months ago, the whole network would just collapse, as none of the instances could keep up with the current volume. That is to say, we have come an extremely long way since last summer alone.

Let me provide some more evidence. Take a look at the Pulse page for the Lemmy backend on GitHub: https://github.com/LemmyNet/lemmy/pulse. As of writing this, Lemmy devs have merged 18 pull requests in the week leading up to this post - that’s an average of 2.5 merged PRs per day. This is extremely good for a project with a small underfunded team.

  1. Rumor: Lemmy devs do not prioritize the important issues

There are two sides to this. First of all, there are endless users who turn to the Lemmy devs with what they believe is the most important issue and should immediately be prioritized - the problem is that almost none of these endless users have the same view of what the most important issue actually is! In that sense, it’s literally impossible to please everybody, because everybody wants different things.

On the other hand, even when Lemmy devs do prioritize things which some users have been desperately asking for, I have on several occasions seen a dismissive response along the lines of “too little too late”. Basically, the demands made are often unrealistic and impossible to meet.

If you are somebody who feels like Lemmy devs are not doing enough, I would ask you to please take a step back, look at the actual contributions which they have made, and consider how you yourself would feel if after making such a massive contribution, you would still need to listen to countless strangers on the internet tell you how you’re not good enough in their opinion.

Conclusion

Lastly, I am very thankful to anybody who took the time to read to the end of this post. Again, my goal is to try and defuse some of the hostility, as well as to put out a message of gratitude and positivity. I am very interested in the success of Lemmy as a whole, and that is much easier to achieve and maintain if we all work together. Thank you, I hope you're doing well, and have a nice weekend!

 

Hey folks

This is a heads up that I will be performing some maintenance and hardware upgrades on our database this Saturday.

We are currently experiencing several spikes throughout the day which cause our database to become overloaded - this results in degraded performance for many users. The spikes are happening due to a combination of continued growth of the database, some expensive periodic scheduled tasks which Lemmy runs, and fluctuating traffic patterns. Some of this can be optimized on the code level in the future, but it seems that the best way to deal with it right now is to add some additional resources to our database server.

I am intending to switch to slightly different hardware in this upgrade, and will be unable to make this switch without downtime, so unfortunately lemm.ee will be unavailable for the duration.

As our database has grown quite a bit, cloning it will most likely take a few hours, so I expect the downtime to last 2-3 hours. Sorry for the inconvenience, I am hopeful that it will be worth it and that this upgrade will significantly reduce some of our recent long page load times!


Edit: upgrade complete!

I have now migrated the lemm.ee database from the original DigitalOcean managed database service to a dedicated server on Hetzner.

As part of this migration, I have also moved all of our Lemmy servers from the DigitalOcean cloud to Hetzner's Cloud. I always want the servers to be as close as possible to the database, in order to keep latencies low. At the same time, I am very interested in having the ability to dynamically spin up and down servers as needed, so a cloud-type solution is really ideal for that. Fortunately, Hetzner allows connecting cloud servers to their dedicated servers through a private network, so we are able to take advantage of a powerful dedicated server for the database, while retaining the flexibility of the cloud approach for the rest of our servers. I'm really happy with the solution now.

In terms of results, I am already seeing far better page load times and far less resource use on the new hardware, so I think the migration has been a success. I will keep monitoring things and tuning as necessary.

view more: next ›