Designing Efficient Heterogeneous Memory Architectures

Recent packaging technologies that enable DRAM chips to be stacked inside the processor package or on top of the processor chip can lower DRAM energy-per-bit costs, provide wider interfaces, and offer higher bandwidth. However, these technologies are limited in capacity and come at a higher price than traditional off-package memories, requiring system designers to balance price, performance, and capacity tradeoffs. The most obvious means to achieve this balance is to employ both on- and off-package memory in a heterogeneous memory architecture. However, designers must then decide whether to deploy the on-package memory as an additional cache-hierarchy level (controlled by hardware or software) or as a memory peer to the off-package DRAM in a NUMA configuration. This article presents a model and analysis of energy, bandwidth, and latency for current and emerging DRAM technologies that enable an exploration of memory hierarchies combining heterogeneous memory technologies with different attributes. The analysis shows that the gap between on- and off-package DRAM technologies is narrower than what is found between cache layers in traditional memory hierarchies. As a result, heterogeneous memory caches must achieve very high hit rates or risk degrading both system energy and bandwidth efficiency.