Hot Chips 2026: Applying High Bandwidth Flash (HBF)

HBF, or High Bandwidth Flash, uses the same flash memory technology we see in SSDs today. Unlike SSDs, HBF is implemented much like HBM (High Bandwidth Memory). HBF cubes sit on the same package as a compute chip, perhaps even next to HBM. HBF’s idea is to offer much higher bandwidth than HBM, while still providing decent memory bandwidth. At Hot Chips 2026 tutorials day, Anurag Agarwal and Radhakrishna Giduthuri’s talk explores how HBF could apply to machine learning workloads. No HBF products exist yet, so the talk focuses on simulations, projections, and how software can adapt to take advantage of HBF.

Even though HBF uses a HBF-like form factor, it’s completely different under the hood. It’s not like Intel’s Optane, which could function as another pool of memory. Instead, HBF is almost like a SSD integrated onto a processor. Software uses DMA to move data between HBF and DRAM. HBF accesses must be done in large, aligned chunks as if it were a mass storage device, rather than system memory. Host software also has to take on SSD controller functions like managing write leveling and ensuring data retention. That means HBF can’t be a plug-and-play solution.

Instead, taking advantage of HBF means formulating a special strategy and implementing it into a runtime. Giduthuri takes vLLM as an example. vLLM typically holds model weights in GPU memory, and is already exploring options to reduce VRAM usage. For example, vLLM is investigating putting model weights in pinned CPU memory provided the host has lots of free memory. While that wouldn’t work for HBF because HBF doesn’t support fine-grained random access, other options might have promise. For example, MoE experts can be stored in HBF. Software can DMA active experts into HBM as needed.

vLLM’s KV cache can also be placed into HBF. However, that may only work well in a sparse attention implementation thatonly reads a subset of tokens off the top of the KV cache for each step. That allows most of the KV cache to sit “cold” in flash, which takes advantage of HBF’s capacity while placing less pressure on HBF’s lower bandwidth. A potential caveat is that the top-k read is scattered, while HBF prefers sequential reads. Perhaps software can get around this by DMA-ing the top-k rows into DRAM as needed.

Another opportunity is using HBF capacity to reduce cross-device communication. Large models are often sharded across multiple GPUs, which results in performance being bound by cross-device scatter and gather operations. Cross-device communication can become a bigger performance barrier than compute throughput or memory bandwidth. HBF can mitigate this by replicating more of a model’s weights across different GPUs. DMA-ing data off flash isn’t cheap, but it’s cheaper than going off-device.

Agarwal went over when HBF makes sense from a cost perspective. Basically, HBF is good if a workload doesn’t reach its bandwidth limits. That applies with smaller models and/or smaller batch sizes. If a workload becomes bandwidth bound, it’s bad for HBF’s cost equation because both cost per capacity and cost per bandwidth factor into final cost. HBF is great for cost per capacity, but is worse in cost per bandwidth compared to HBM.

He also discussed using HBM to cache hot experts, but that also seems like a difficult solution. Caching needs to work out well, or HBF bandwidth can throw a wrench into the works for the cost-per-token equation.

Final Words

HBF may alleviate the DRAM capacity problem to some extent, but the software challenges feel immense. Handling HBF sounds a lot like working with a low level disk access API, like using FILE_FLAG_NO_BUFFERING in Windows or O_DIRECT in Linux. Software has to carry out accesses in large, aligned chunks rather than freely addressing storage with byte-level granularity. Modifying a single byte can mean reading a large 64 KB block into DRAM, modifying that block, and writing the whole block back to flash. That’s more like working with a block storage device than working with memory. A software framework meant to work with a regular DRAM-based system will need massive changes to leverage HBF. Moving to a different framework will mean re-doing the work needed to take advantage of HBF.

I would go as far as saying that the effort required to leverage HBF doesn’t seem far off what would be required to straight up reduce DRAM usage by streaming model weights off a SSD. Taking advantage of SSD actually seems easier. The OS kernel can abstract away the difficulty of doing block-aligned accesses if you don’t use FILE_FLAG_NO_BUFFERING or O_DIRECT,. Buffering in the kernel will let software arbitrarily seek and carry out byte-level read/write operations. It’ll also act as a cache, naturally insulating software from flash memory inefficiencies. While not mentioned in the talk, I wonder if existing attempts to stream model weights off SSDs can be applied to HBF. Or, if the software challenges associated with using HBF prove too steep and prevent its adoption. I guess we’ll see when/if HBF products hit the market. I want to see something that’ll alleviate the current DRAM shortage, but I’m not sure if HBF is it.

Generated by RSStT. The copyright belongs to the original author.

Source