Basically the title.

For me, a dense model around 20b, with an additional engram table of 10-20b, or a sparse MoE around 80b would be amazing.

you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 1 week ago* (2 children)

Oh hell yeah

I'm on my 1070 8gb copium so we're twinsies

Maybe they release a huge moe model with non-uniform quants and then we can get even more quality on those cute little cards

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 2 days ago (1 child)

    How's your experience with the 1070?

  • source
  • parent
  • hideshow 1 child comment
  • Mixed?

    It's the lower bound of where LLMs are helpful. Anything less and the models would be either too slow to work or too dumb. With the 1070 they are already painfully slow but at least they're still good enough to do some menial tasks.

    A good harness for web search etc does a lot though.

  • source
  • parent