1247
"LLMs can do anything" except make positive cashflow. (europe.pub)
submitted 5 days ago by to c/funny@sh.itjust.works
135 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] 7 points 5 days ago (19 children)

turn the models into silicone

The models are data, and silicone is rubber. So... I don't know what you're proposing here.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 19 child comments
    •
  • [–] 3 points 5 days ago (13 children)

    Might mean "silicon", but might want to bang the LLM. Hard to know.

    Words have meaning...

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 13 child comments
    •
  • [–] -1 points 5 days ago (12 children)

    Even with "silicon" doesn't really make sense. The models already live on silicon, they're stored in the datacenters needed to run them. It's not like you can make them into some chip and magically not have to use the datacenter.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 12 child comments
    •
  • [–] 4 points 4 days ago (11 children)

    Gonna take GPS example to explain general processors vs specific ones.

    GPS works by doing some simple math around triangulation from satalites. When they first came out there were no GPS chips. It was done by a general processors.

    Now they can do it, but because they can do Amy Hong they're not as fast as newer GPS chips.

    How they work is the silicon can only handle the GPS calculations. But because of that, it can do it way faster.

    LLMs run on matrix math and probabilities. We can 100% come up with specialised silicon to handle this math. Intact, thats what a lot of AI specialised chips are.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 11 child comments
    •
  • [–] 3 points 4 days ago (1 child)

    NPUs are already a thing, specialized cpus for nerual networks. The problem is the models are just databases, and you need lots of fast memory in order to feed the NPUs data, and thats the current bottleneck.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 4 days ago (5 children)

    They aren't comparable. GPS is deterministic and simple so as you said it could be reduced to an FPGA or ASIC.

    LLMs are partially matrix math and probabilities but they're also more complex than that and require a large amount of RAM to run even the first time. Each query added to the context increases the RAM needed further.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 1 point 4 days ago (1 child)

    I dont believe the memory, both ram and storage combine, are what's causing data centres to gobble up power and associated resources (electricity, cooling,...).

    Its the GPUs.

    I'm not counting on a future with LLMs by any means, but there's tons of efficiencies we can do to make the cost go down. Were just not doing it.

    We saw the same thing with crypto mining. It started on GPUs but then speficially designed silicon (asyc miners) were vastly more efficient than a GPU rig.

    It still needed memory yes, but again, thats not the constraint.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 4 days ago

    For crypto mining, the speed of computation before the next block was mined was the constraint so GPUs were being used for parallel computation before ASIC miners were built. This is a great example of where ASICs shine: deterministic simple algorithms that don't require much if any I/O.

    LLMs have a minimum RAM requirement in order to even load the model so even if you built a dedicated LLM card with an ASIC, you would still need at least that amount of RAM on the card. You'd probably want more RAM in order to store context because that takes RAM as well. At that point you pretty much have a GPU with an ASIC instead of more general compute units and no video outputs.

    You might get some improvements by using an ASIC for LLMs but not nearly as much as when ASICs started being used for mining.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 4 days ago (2 children)

    LLMs are also deterministic and the weights for a model could be 'hard-coded' into a chip, no?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 4 days ago (1 child)

    They're actually not because they have randomness built in. Since we are working with probabilities, it won't always pick the next token that has the highest probability and the randomness can be tuned via a "temperature" setting to make it more or less likely that it will choose the most probable token.

    The weights for the model could be stored in a firmware chip but you still need RAM because it pulls all the weights into RAM in order to perform the calculations.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 3 days ago* (last edited 3 days ago)

    The LLM itself is deterministic. It outputs a vector that we interpret as a probability distribution over the set of tokens. It's the program using the output of the LLM, such as a chatbot program, that selects an individual token using (or not using) these vector elements as weights.

    The weights do not change once the model is trained. This is why I am suggesting they could be incorporated directly into the structure of an ASIC for a specific model, rather than storing them in memory.

    Edit: Of course another major factor could be that the models are just to big to be wholly implemented in a single IC by any currently existing manufacturer.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 4 days ago (2 children)

    LLMs run on matrix math and probabilities

    Yes, and the silicon to handle that math already exists. It's the "GPUs" being pumped out by nVidia. Those are basically now highly specialized matrix math machines.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 4 days ago (1 child)

    Then why do NPUs exist?

    What I'm talking about exists on the smaller scale. Just a matter of time before its done on a larger.

    A general purpose GPU still contains silicon for display, and video en/decoding. Those don't contribute in any meainiful way.

    Not to mention it doesn't cover all of matrix math for LLM needs....

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 4 days ago

    NPUs are low-power hardware devices for running very small AI models.

    So-called "GPU"s are the devices used for similar things for the very large models that are the "hotness" right now.

    A general purpose GPU still contains silicon for display, and video en/decoding

    No it doesn't. The modern datacenter-style "GPU"s that nVidia has been making and selling for use in AI datacenters have stripped out all the video related elements. They're purpose built for doing only AI work these days. They can't even do standard matrix multiplication work anymore because they're so narrowly designed for AI models where they use a specialized form of lossy matrix multiplication.

    The name has stuck around, but the things nVidia has been building and selling to the AI companies no longer has anything to do with graphics.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 1 point 4 days ago (2 children)

    Fixed it. We don't use this silly silicon word here. I picked the wrong one. We call it Silicium Sodium is Natrium, Potassium is Kalium, Aluminum is Aluminum etc.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 4 days ago (1 child)

    Interesting, your words match the periodic table better.

    But, even with that, I'm not sure what you're proposing. The models (data) are already stored in silicon. They're on SSDs or HDDs inside data centers. But, to use those models you need to do inference which (at least for the big American models) requires a data center. It's not like you could do it on some gadget you carry in your pocket.

    There are much smaller models that allow you to do that. But, because they're smaller models they aren't as accurate at the word generation game.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 3 days ago

    I mean this: https://en.wikipedia.org/wiki/Application-specific_integrated_circuit

    There are companies working in this direction already. And regarding model size the Chinese models aren't as large as the American ones. Not on the same level but still usablzand if I had one in a little box in my office I could run it 24/7 with lightning speed which would probably make up for the model performance difference.

    It's already possible with GDX Spark and similar hardware but it's relatively slow, can't run the full size models and pretty expensive.

  • permalink
  • fedilink
  • source
  • parent