I believe you've answered your own question.
Lemmy isn't Marxist-only. The majority of Lemmy users are what the more vocal Lemmygrad and Hexbear users deride as "libs." As a thought experiment, imagine that you are one of us for a moment and then browse Local on one of those.
Exhibit #1 why Hexbear/Lemmygrad are unpopular: this guy