the classic language models before LLMs existed

Which models are you referring to here? Earlier versions of GPT (like GPT-2)?

There are small LLMs like Phi 4 Mini that you could look at. Those were part of the "small language model" craze (SLMs).

If you want meaningful commit messages though, nothing beats handwritten messages. Commit frequently and the messages should be short and easy to come up with.

https://victorzhou.com/blog/better-profanity-detection-with-scikit-learn/

That might be of interest to you then? Its a small model trained for profanity detection, might work for your use case. Dunno if it can be used for generation though.

Otherwise, there were 1GB gemma models, you can run them on CPU.

Its small in the sense it'll run on basically any GPU from the last decade, rather than high end or specialised GPUs. And there is a 1GB gemma4 model, which is meant for CPU/phones.