CS 1104 Help Session I Caches

Colin Tan, ctank@comp.nus.edu.sg S15-04-15

Topics for Today

Session I - Cache, Session II - Virtual Memory Basically we will pick off important topics and elaborate on them. Also will have simple problems to solve
Please have your calculator, paper and pen ready!

Help session notes are available at:

http://www.comp.nus.edu.sg/~ctank

Memory Hierarchy
Motivation
Not all memory is created equal
Cheap Memory => Slow Fast Memory => Expensive
DRAM, 70 ns access time, $5/MByte SRAM, 8 ns access time, $100/MByte

The purpose of memory hierarchy is that it allows us to have huge amounts of cheap that operate close to the speed of expensive but fast memory.

How do we do that?
We can achieve this by using locality. Locality is the behavior of programs such that:
Data or instructions that was recently accessed is likely to be accessed in the near future (Temporal Locality) The variable x in this code fragment will have temporal locality for(int i=0; i<100; i++) y= y+x The neighbors of data or instructions that was recently accessed are likely to also be accessed (Spatial Locality) Program execution tends to be sequential, so if an instruction was just executed, it is likely that the next instruction next to it will also be executed.

Caches
Locality means that only small portions of main memory will be used in the near future. We can create a small memory system with fast but expensive devices to store these portions of main memory. We can then access these portions of memory very quickly. This is the concept behind caches.

How Do Caches Help?

The average time to access memory (AMAT) is given by:
AMAT = Tcache + miss_rate x Tmemory
Tcache = Time to read the cache (8ns for SRAM cache)

Tmemory = Time to read main memory (70ns for DRAM) miss_rate = Probability of not finding what we want in the cache.

Because of locality, miss_rate is very small

Typically about 3% to 5%.

On the average, time to access memory is very close to that of the cache (8ns) rather than that of the main memory (70ns).

How do Caches Help?

Yet at the same time, we have the benefit of being able to have large amounts of memory
This is because most of our memory is cheap DRAM!

The net effect is that caches allow us to have huge amounts of cheap, slow memory, yet have access times of fast, expensive memory.

Basics of Addressing
Memory Addresses
Memory is a huge array of data. Just like arrays, we must specify the index of the piece of data that we want, so that we can get it out of the array. In memory systems, this index is called an Address.

Where do Addresses Come From?

For instruction fetches, the address of the instruction (i.e. the location in the memory where the instruction is in) comes from the Program Counter. For data fetches, the address comes from the ALU stage of the pipeline whenever we do a lw or sw operation. In the MIPS architecture, addresses are 32bit numbers.

Cache Architecture
Caches consist of blocks (or lines). Each block stores data from memory:
Block

Block allocation problem:

Given data from an address A, how do we decide which block of cache its data should go to?

The Block Allocation Problem

3 possible solutions:
Data from each address A will go to to a fixed block.
Direct Mapped Cache

Data from each address A will go to any block.

Fully associative cache

Data from address A will go to a fix set of blocks.

Data may be put into any block within a set. Set associative cache.

Direct Mapped Caches

The value of a portion of memory address is used to decide which block to send the data to:
Address A Tag Block Index Block Offset Byte Offset

The Block Index portion is used to decide which block data from this address should go to.

Example
The number of bits in the block index is log2N, where N is the total number of blocks. For a 4-block cache, the block index portion of the address will be 2 bits, and these 2 bits can take on the value of 00, 01, 10 or 11. The exact value of these 2 bits will determine which block the data for that address will go to.

Example
The value of the two block index bits will determine which block the data will go to, following the scheme shown below:
Cache
00 01 10 11

Solving Direct-Mapped Cache Problems

Question 7.7
Basic formula:
Blk_Addr = floor(word_address/words_per_block) mod N
N here is the total number of blocks in the cache This is the mathematical version of taking the value of the Block Index bits from the address.

A Complication: Multiple Word Blocks

Single word blocks do not support spatial locality
Spatial locality: Likelihood of accessing neighbor of a piece of data that was just accessed is high. But with single word blocks, none of the neighbors are in cache!
All accesses to neighbors that were not accessed before will miss!

An Example Question 7.8

Accessing Individual Words

In our example, each block has 4 words. But we always access memory 1 word at a time! (e.g. lw) Use the Block Offset to specify which of the 4 words in a block we want to read:
Address A

Tag

Block Index Block Offset

Byte Offset

The Block Offset

Number of block offset bits = log2M, where M is the number of words per block. For our example, M=4. So number of block offset bits is 2. These two bits can take on the values of 00, 01, 10 and 11. Note that for single word blocks, the number of block offset bits is log2 1, which is 0. I.e. There are no block offset bits for single-word blocks. These values determine exactly which word within a block address A is referring to:

The Block Offset

Word 00
00 01 10 11

Word 01

Word 10

Word 11

4 block cache, 4 words per block

The value of the 2 block offset bits (see previous slide) determine if our address A is referring to word00, word01, word10 or word11.

Who am I? Purpose of the Tag

Many different addresses may map to the same block: e.g. (Block Index portions shown highlighted)
01000 00010010
01010 00010010

00000000
00000000

00
00

11011

00010010

00000000

00

All 3 addresses are different, but all map to block 00010010

Disambiguation
We need a way to disambiguate the situation
Otherwise how do we know that the data in block x actually comes from address A and not from another address A that has the same block index bit value?

The portion of the address A to the left of the Block Index can be used for disambiguation. This portion is called the tag, and the tag for address A is stored in the cache together with address A data.

The Tag
Tag
00 01 10 11

Word 00

Word 01

Word 10

Word 11

When we access the cache, the Tag portion and Block Index portions of address A are extracted. The Block Index portion will tell the cache controller which block of cache to look at. The Tag portion is compared against the tag stored in the block. If the tags match, we have a cache hit. The data is read from the cache.

Accessing Individual Bytes

Address A

Tag

Block Index Block Offset

Byte Offset

MIPS addresses are byte addresses, and actually index individual bytes rather than words. Each MIPS word consists of 4 bytes. The byte offset tells us exactly which byte within a word we are referring to.

Advantages & Disadvantages of Direct Mapped Caches

Advantages:
Simple to implement Fast performance
Less time to detect a cache hit => less time to get data from the cache => faster performance

Disadvantages
Poor temporal locality.
Many addresses may map to the same block. The next time address A is accessed, it may have been replaced by the contents of address A.

Improving Temporal Locality The Fully Associative Cache

In the fully associative cache, data from an address A can go to any block in cache.
In practice, data will go into the first available cache block. When the cache is full, a replacement policy is invoked to choose which block of cache to throw out.

Example
Question 7.22

Searching the Cache

In the fully associative cache, an address A is split into the following parts:
Address A

Tag

Block Offset

Byte Offset

The cache controller will search the entire cache to see if it can find a block with the same tag value as the tag portion of A. If it can find such a block, we have a cache hit, and the controller reads the data from the cache.

Advantages and Disadvantages Fully Associative Cache

Good temporal locality properties
Flexible block placement allows smart replacement policies such that blocks that are likely to be referenced again will not be replaced. E.g. LRU, LFU.

Disadvantages
Complex and too expensive for large caches
Each block needs a comparator to check the tag. With 8192 blocks, we need 8192 comparators!

A Compromise Set Associative Caches

Represents a compromise between directmapped and fully associative caches. Cache is divided into sets of blocks. An address A is mapped directly to a set using a similar scheme as for direct mapped caches. Once the set has been determined, the data from A may be stored in any block within a set - Fully associative within a set!

Set Associative Cache

An n-way set associative cache will have n blocks per set. For example, for a 16-block cache that is implemented as a 2-way set associative cache, each set has 2 blocks, and we have a total of 8 sets.

Set Associative Cache

Block 0 Block 1

Set 000 Set 001 Set 010 Set 011 Set 100 Set 101 Set 110 Set 111

An address A will be divided into:

Address A Tag Set Index Block Offset Byte Offset

Accessing a Set Associative Cache

Address A
Tag Set Index Block Offset Byte Offset

The Set Index portion of address A is extracted. This is used to index the sets (i.e. If the Set Index portion is 010, then this address is mapped to Set 010). The tag portion of A is extracted and compared against the tags stored in Block 0 and Block 1 of Set 010.

Accessing a Set Associative Cache

If a match is made either in Block 0 or Block 1 of Set 010, then we have a cache hit, and the data for A is read from the cache block. If we have a miss, then the data for A is fetched from main memory, and placed in the first available block in Set 010. If no blocks are available, a replacement policy is invoked to choose a block to replace.

Example
Question 7.20
Basic formula:
Blk_Addr = floor(word_address/words_per_block) mod N
Here N is the number of sets, NOT NUMBER OF BLOCKS! This is the mathematical version of taking the value of the Block Index bits from the address.

Multi-block Set vs. Multi-word blocks?

Confusion often arises over multi-block sets (or n-way set associative) vs. multi-word blocks. Each block in a set can itself have multiple words, like the blocks in question 7.8. Each block will also have its own tag.

Advantages and Disadvantages Set Associative Cache

Advantages
Almost as simple to build as a direct-mapped cache. Only n comparators are needed for an n-way set associative cache. For 2-way set-associative, only 2 comparators are needed to compare tags. Supports temporal locality by having full associativity within a set.

Advantages and Disadvantages Set Associative Cache

Disadvantages
Not as good as fully-associative cache in supporting temporal locality. For LRU schemes, because of small associativity, actually possible to have 0% hit rate for temporally local data. E.g. If our accesses are A1 A2 A3 A1 A2 A3, and if A1, A2 and A3 map to the same 2-way set, then hit rate is 0% as subsequent accesses replace previous accesses in the LRU scheme.

Writing to Cache
Remember that data in cache is merely a copy of data in main memory. When data that is stored in a cache block is modified (e.g. when doing a sw to address A), then the copy in cache will become inconsistent with the copy in memory. Need a way to maintain consistency.

Memory/Cache Consistency 2 solutions

Write-through cache
In the write-through cache, consistency between cache data and memory data is maintained by updating both main memory and cache. This is very slow
Must wait for both cache and memory writes to complete before CPU can proceed. Memory writes are very slow!

Memory/Cache Consistency 2 solutions

Write-back Cache
Only the cache copy of data is updated. When the data in a block is updated, a special flag called the dirty bit will be set to indicate that the cache copy is now inconsistent with the memory copy. If the block is chosen for replacement (either by replacement policy or because another address A maps to the same block), then the memory copy is updated if the dirty bit is set. If dirtybit is not set, the block is simply replaced.

Nitty-Gritty Use of Addresses by Cache

Addresses are used to access cache. For Direct Mapped Cache:
Address A Tag Block Index Block Offset Byte Offset

Nitty-Gritty Use of Addresses by Cache

Direct Mapped Cache
Address A

Tag

Block Index Block Offset

Byte Offset

Number of Byte Offset Bits B= log2(number of bytes per word) On MIPS this is usually 2 bits Number of Block Offset Bits W= log2(Number of words per block) 0 bits for 1-word blocks Number of Block Index Bits I = log2(Number of blocks) Number of tag bits = address_length - B - W- I address_length is 32 bits on MIPS

Nitty-Gritty Use of Addresses by Cache

Fully Associative Cache
Address A Tag Block Offset Byte Offset

Number of Byte Offset Bits B= log2(number of bytes per word) On MIPS this is usually 2 bits Number of Block Offset Bits W= log2(Number of words per block) 0 bits for 1-word blocks Number of tag bits = address_length - B - W address_length is 32 bits on MIPS Note that there are no index bits for fully associative caches.

Nitty-Gritty Use of Addresses by Cache

Set-Associative Cache
Address A Tag Set Index Block Offset Byte Offset

Number of Byte Offset Bits B= log2(number of bytes per word) On MIPS this is usually 2 bits Number of Block Offset Bits W= log2(Number of words per block) 0 bits for 1-word blocks Number of Set Index Bits S = log2(Number of sets) Number of tag bits = address_length - B - W- S address_length is 32 bits on MIPS

Example
A cache built for the MIPS architecture has a total size 128 KB. Find the total number of tag, set index, block index, block offset, and byte offset bits for a given address A for each of the following cache architectures:
Direct Mapped, 1 word per block. Direct Mapped, 8 words per block Fully associative, 2 words per block 2-way set associative, 4 words per block

Example
Basic things you first need to work out:
What types of information do I need to determine for each cache architecture?
E.g. for set-associative, need to determine byte-offset, block offset, set index and tag bits.

What is the cache size in terms of words? What is the total number of blocks that we would have, or the total number of sets?
This will give us the number of index bits.

Any other important information?

Nitty-Gritty Cache Housekeeping Flags

Other than the data and tag bits, cache blocks need to store housekeeping flags. The dirty bit (D) we saw earlier is an example. Other bits include:
Valid bit (V)
When a cache first starts up, the tag and data bits are random. It is possible to have a cache hit because the tag from an address may match a random number in the tag field of a block. But the data is random in invalid! The Valid bit is normally off, and will be set when valid data is written to a block.

Nitty-Gritty Cache Housekeeping Flags

Use Bit (U)
This is used by the LRU replacement algorithm to determine which block is LRU. Present only in fully-associative and set-associative caches using LRU replacement policies.

A cache may thus look as complicated as this: V D U Tag Word 00 Word 01 Word 10

Word 11

Total Number of Bits in Cache?

When we speak of cache size, we normally refer to how many bytes of main memory data the cache can hold
E.g. a 64KB cache can hold up to 64KB of main memory data

But as we can see, data is not the only thing stored in a cache block.
We also have the tag and housekeeping flags!

Thus the total number of bits needed to implement a cache can be much bigger than the specified cache size!

Example
We want to implement a 256KB write-back cache on the MIPS architecture. The cache will be 4-way set associative, with 4 word blocks. The LRU replacement policy will be used. Find the total number of bits of SRAM required to implement this cache.

Example
Analysis
What housekeeping flags will be needed? What is the size of the data portion of each block? What is the number of blocks? What is the number of sets? What is the number of tag bits?
What is the number of byte offset, block offset and set index bits required?

Based on this analysis, you should be able to get the answer.

Summary
Caches
Make use of locality to make it possible to have small amounts of fast expensive memory hold a copy of main memory data that is likely to be accessed soon. Allows fast access of huge amounts of memory.

Cache types
Direct Mapped
Simple, fast Poor temporal locality

Summary
Fully Associative
Flexibility of block placement allows smart replacement algorithms that promotes temporal locality. Expensive, slow.

Set Associative
Simpler to build than fully associative, yet gives good temporal locality through flexible placement of blocks (just like fully-associative). Limited associativity can sometimes give poor performance

Summary
Writing policies
Write-through
Simple to implement Slow

Write-back
Fast Difficult to implement

Housekeeping flags
Need extra info for the running of the cache

Total Cache Sizes vs. Cache Sizes

Not the same thing!