158
Is anyone self-hosting their own AI? (europe.pub)
submitted 1 week ago by to c/selfhosted@lemmy.world
190 comments fedilink hide all child comments
 

Does anyone run their own AI service?

If so,

Which one?

What hardware?

Do you have a way to access it from outside or from a phone?

And, if you're open to sharing, what do you do with it?

I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don't absolutely need specialized hardware to run and perform well enough for many purposes.

I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I'm not comfortable using them to help me with web searches and the such, because there's no way that all this information won't be going into a profile. This makes me want to have my own "helper" service.

you are viewing a single comment's thread
view the rest of the comments
[–] 8 points 6 days ago (6 children)

I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it's been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.

  • permalink
  • fedilink
  • source
  • hideshow 6 child comments
    •
  • [–] 4 points 6 days ago* (2 children)

    I have nearly the same setup, except OpenCode for the harness. I run 200k context for anything that doesn’t need vision, or 150k otherwise. Typically get about 30 t/s decode With MTP, starts at close to 800 t/s for prefill and drops to the 200s for a long context. I’d imagine higher decode speeds are possible with some tinkering, but I haven’t gotten around to it.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 5 days ago*

    I get about 45 t/s after some tweaking, getting the thermals under control helped a lot; mainly by reducing the power consumption to 261 W (and downclocking a little bit).

    What vision model do you use?

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 6 days ago (2 children)

    How much electricity does it burn? And how much does it cost to let it run overnight? I feel like this is the same problem that the companies have, its way too expensive to use it on a lots of things and it is way cheaper to use it on someone else's server..

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •