LocalLLaMA
Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.
Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.
As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.
Rules:
Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.
Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.
Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.
Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.
view the rest of the comments
llama-server has router mode built-in; you don't need separate software to swap models. I keep a list of my models in a presets.ini file.
Yes but llama server does not unload models to fit a different model. At least, didn't managed to have just llama.cpp swap between models that fill up my vram... The second one would fail load.
You can set a maximum number of slots and it will automatically unload to load a different model, I just set mine to 1 slot so there's only 1 model loaded at a time
It's pretty limited, with swap I can create groups of models that can live together, much more flexible
Ah. I POST to '/models/unload' when I switch models as part of my tooling -- if llama-swap is doing that for you, then I get why you'd want it.
Yes, llama swap also give you a pretty web based statistics of all the calls and runs for every model with t/s and more statistics.
It's pretty neat... You can also load and unload models manually, define groups for models that fit together in vram and so on.
It gives you that automation that people coming from ollama are used to.
under the hood all it does is running llama serve. You convert your models.ini to a yaml file 1:1 (plus a few more flexibility).
This is the way 👆