Taalas HC1

“Given that it's ~50k tokens per second, even a lesser model opens up an entire universe of use-cases”

“Those on-device weights are horribly outdated, underperform for their size, and cannot be updated to use a different model.”

“That saving is sitting right there, today, ready for anyone who is willing to say "this is the model we're freezing on for a couple of years"”

“I take 200W for 17,000 tokens/sec any day, even just for Llama 3.1 8B.”

“The chip architecture is interesting, the first (n) layers are the sane, so you can change architecture by adding (m) layers.”

“the model will be hopelessly outdated (with the current improvement rates)”

“HC1 was a relatively low budget proof of concept. The team that built is working on a better implementation.”

“A card with two of them chips and you can already have a dense Qwen at staggering speeds.”

“I'd buy one right now, even knowing that I'm "stuck" with whatever model of the day is.”

“Those saying the model sucks are completely missing the point: it was a proof-of-concept.”

“delivering 17k tokens per second per user on Llama 3.1 8B model”

“has the massive and enabling advantages of running NNs very fast and very energy efficiently”

“using an ASIC with weights in ROM means you can’t update the model (weights or architecture) without replacing the entire chip”

“The point is inference speed and power, making previously impossible local inference possible.”

“this could easily become a qualitatively different form of autocomplete/text suggestion”

“Frankly, I don't see the use-case for this tech. It's too expensive and too inflexible.”

“Their HC1 die is already near the maximum size that conventional lithography can expose.”

“A model can't be updated, and a chip that is only relevant for 6 months at max?”

“The tokens per second performance numbers coming from Cerebrus/Talas are several orders of magnitude higher than models running on GPUs”

“the amount of compute you can do in parallel one one of those chips is amazing”

“The actually impressive and novel thing is that Taalas’ve managed to automate that process”

“the HW is achieved, working, online, available... and amazing”

“Imagine when we can roar along at that speed, low power.”

“Having lightning-speed, local inference of a super high-quality model would be incredible.”

“running on specialized silicon implementing just it. So has super low energy use and blazing speed.”

“So clearly hardware LLMs are the future, and the cost will be drastically reduced.”

“They've made a hardware LLM that reaches over 14k TPS, and you can try it here”

“a Taalas chip churning out 17k tokens/sec could unlock”

“now it turns out it does and there's a company making it fast and robust!”

“It will cost way more for similar performance as what you would get with a gpu.”

“10x faster and 20x cheaper than traditional GPU inference”

“Taalas requires more silicon which will always cost more than storing weights in DRAM.”

“might end up being fast but inflexible -- good for commodity tasks, not for anything that needs customization per customer.”

“The Taalas approach is much more expensive than the NPU that phones already have.”

“Talas promises a 10x higher throughtput, being 10x cheaper and using 10x less electricity.”

“the custom hardware they are producing and its performance, that is whats impressive”

“Yea its fast af but very quickly loses context/hallucinates from my own tests with large chunks of text”

“the Llama 3.1 8B model Taalas uses predictably results in incorrect answers, hallucinations, poor reliability as a chatbot”

“I think these will sell, even on 6nm process, and the first thing I'd sell them to do is speculative decoding”

“potentially a lot cheaper than Nvidia for those purposes”