And a small cluster like Alibaba used to train Qwen 2.5 is basically a drop in the bucket.
The hoard of GPUs Meta, Microsoft/OpenAI, and especially X have are apparently being used extremely inefficiently, or perhaps mostly not used to train AI at all, but do regular ad/engagement optimization stuff.
Ah yeah those were the good old days when vendors were free to do that, before AMD/Nvidia restricted them. It wasn’t even that long ago, I remember some AMD 7970s being double VRAM.
And, again, I’d like to point out how insane this restriction is for AMD given their market struggles…
Strix Halo is your unicorn, idle power should be very low (assuming AMD VCE is OK over quicksync)
Just that bursts of inference for a small model on a phone or even a desktop is less power hungry than a huge model on A100s/H100s servers. The hardware is already spun up anyway, and (even with the efficiency advantage of batching) Nvidia runs their cloud GPUs in crazy inefficient voltages/power bands just to get more raw performance per chip and squeak out more interactive gains, while phones and such run at extremely efficient voltages.
There are also lots of tricks that can help “local” models like speculative decoding or (theoretically) bitnet models that aren’t great for cloud usage.
Also… GPT-4 is very inefficient. Open 32B models are almost matching it at a fraction of the power usage and cost, even in servers. OpenAI kind of sucks now, but the larger public hasn’t caught on yet.
Oh also you might look at Strix Halo from AMD in 2025?
Its IGP is beefy enough for LLMs, and it will be WAY lower power than any dGPU setup, with enough vram to be “sloppy” and run stuff in parallel with a good LLM.
You could get that with 2x B580s in a single server I guess, though yoi could have already done that with the A770s.
Also “ridiculously” is relative lol.
The Llm/workstation crowd would buy a 48GB 4060 without even blinking, if that were possible. These workloads are basically completely vram constrained.
Like the 3060? And 4060 TI?
Its ostensibly because they’re “too powerful” for their vram to be cut in half (so 6GB on the 3060 and 8GB on the 4060 TI), but yes, more generally speaking these are sweetspot for vram heavy workstation/compute workloads. Local LLMs are just the most recent one.
Nvidia cuts vram at the high end to protect their server/workstation cards, AMD does it… Just because?
deleted by creator
There’s no way. I think he’s legit too old, and he only needs 1 term to pardon himself and everyone he wants.
Are the double capacity GDDR6X ICs being sold yet? I thought they weren’t and “double” cards like the 4060 TI were using 2 chips per channel.
I mean, if you even have to go into the bios or dip into the mechanics of drive letters and formatting, you have already lost most people.
I feel like I would reach for the used market over a 7600 based on what I read. Even if there were power constraints, one could power cap a 6000 series card.
…Hence I got a 3090 when the 4000/7000 series was long out.
Well AMD won’t do it ostensibly because they have a high mem workstation card market to protect, but the running joke is they only sell like a dozen of those a month, lol.
Intel literally had nothing to lose though… I don’t get it. And yes, this would be a very cheap thing for them to try, just a new PCB (and firmware?) which they can absolutely afford.
Again, this is awesome but probably not the main factor, as most people dont know how to install an OS.
Its weird that Intel/AMD seem so disinterested in the LLM self hosting market. I get its not massive, but it seems way big enough for niche SKUs like they were making for blockchain, and they are otherwise tripping over themselves to brand everything with AI.
If there even is a 780. Rumor is it was canceled.
I am skeptical of this because the majority of people do not know how to install an OS.
I think its people not using Windows at all. Colloquially, I know young people that basically only know how to use mobile interfaces and tablets.
Ah… Yeah, I’d wager the bulk is going to phones and tablets, and that should be extremely telling for anyone at Microsoft trying to enshittify 11.
The American way is to bail them out financially, then re-elect them.