High-Bandwidth Memory Changes the Cost of Moving Data
High-Bandwidth Memory Changes the Cost of Moving Data
Accelerators gain speed when their most frequently accessed data stays close to the compute die.
High-bandwidth memory stacks memory dies and connects them through a wide interface inside the package. The short, wide link can move a great deal of data in parallel.
Package area, memory capacity and cooling limit what can fit. Workloads that already spend most of their time on arithmetic may see less improvement than those waiting for data.
A benchmark should report model size, batch size and precision along with throughput. These conditions determine how often the processor needs to fetch data and whether memory bandwidth is the bottleneck.
