you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 1 week ago

The article from Anthropic describes how a small agentic model~15-50gb can be distributed on an inference server hosted across a network of 10-20 IoT devices like smart-TVs, which can have up to 4gb RAM--higher end IoT appliances can also have 1-4GB RAM to hold a meaningful shard of a 30B model.

  • source
  • parent