Basically the title.

For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.

you are viewing a single comment's thread
view the rest of the comments
[–] 12 points 1 week ago (4 children)

gemma 26b-a4b was fantastic, anything in that area is ideal

running my tiny rx6600 8gb vram

  • source
  • hideshow 4 child comments
  • [–] 3 points 1 week ago* (2 children)

    Oh hell yeah

    I'm on my 1070 8gb copium so we're twinsies

    Maybe they release a huge moe model with non-uniform quants and then we can get even more quality on those cute little cards

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 1 day ago (1 child)

    How's your experience with the 1070?

  • source
  • parent
  • hideshow 1 child comment
  • Mixed?

    It's the lower bound of where LLMs are helpful. Anything less and the models would be either too slow to work or too dumb. With the 1070 they are already painfully slow but at least they're still good enough to do some menial tasks.

    A good harness for web search etc does a lot though.

  • source
  • parent
  • [–] 1 point 1 week ago

    I have an M4 Pro 24GB and I still use Gemma 26B-A4B as my primary model. Beats every other model I’ve ever tried locally on most stuff except coding and tool use, so a Qwen 4 at this size would be super interesting

  • source
  • parent