content_image/994054.jpg
In most real-world situations, a fair number of cache lines exist in multiple L3s underneath a specific L4. The L4 does not contain the same line multiple times, but rather once with an indication of all the cores that have a copy of that line. As such, 672 MB of inclusive L4 can easily cover 768 MB of underlying L3 caches. 

GLYPH<SM590000> Main storage has up to 8 TB addressable memory per CPC drawer, which uses 25 DIMMs. A four-CPC drawer system can have up to 32 TB of main storage. 

## Considerations 

Cache sizes are being limited by ever-diminishing cycle times because they must respond quickly without creating bottlenecks. Access to large caches costs more cycles. Instruction and data cache (L1) sizes must be limited because larger distances must be traveled to reach long cache lines. This L1 access time generally occurs in one cycle, which prevents increased latency. 

Also, the distance to remote caches as seen from the microprocessor becomes a significant factor. An example is an L4 cache that is not on the microprocessor (and might not even be in the same CPC drawer). Although the L4 cache is rather large, several cycles are needed to travel the distance to the cache. The node-cache topology of z14 servers is shown in Figure 3-2. 

<img src='content_image/994054.jpg'>

Although large caches mean increased access latency, the new technology of CMOS 14S0 (14 nm chip lithography) and the lower cycle time allows z14 servers to increase the size of cache levels (L1, L2, and L3) within the PU chip by using denser packaging. This design reduces traffic to and from the shared L4 cache, which is on another chip (SC chip). Only when a cache miss occurs in L1, L2, or L3 is a request sent to L4. L4 is the coherence manager, which means that all memory fetches must be in the L4 cache before that data can be used by the processor. However, in the z14 cache design, some lines of the L3 cache are not included in the L4 cache. 

