> so you had to have lots of operand re-use to not be memory-bound Looking at Nv... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

musicale on March 6, 2025 | parent | context | favorite | on: Apple M3 Ultra

> so you had to have lots of operand re-use to not be memory-bound

Looking at Nvidia's spec sheet, an H100 SXM can do 989 tf32 teraflops (or 67 non-tensor core fp32 teraflops?) and 3.35 TB/s memory (HBM) bandwidth, so ... similar problem?

pklausler on March 6, 2025 [–]

There is caching today.

ryao on March 6, 2025 | [–]

The cache hitrate is effectively 0 for LLMs since the datasets are so huge.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact