When do non-temporal stores pay off?

1Two paths to memory

Figure 1 shows the two routes a store can take. A regular store needs the line in cache first, so writing a line the core does not have costs a read from DRAM (the read-for-ownership¹) and, later, a write back. A non-temporal store goes to a write-combining buffer and leaves as one full-line write.

2A simple model

For a fill of $𝑁$ bytes that misses every cache level, with memory bandwidth $𝐵$, the regular path moves each byte twice and the non-temporal path once:

Equation (1) is an upper bound on the speedup for a memory-bound fill. It assumes the buffer is much larger than the last-level cache; below that, the regular path keeps the data in cache and wins. Averaged over a run of fills, the measured speedup $\overline{𝑆}$ lands somewhere in $(1,2]$.

3In code

In C, the AVX2 intrinsic is _mm256_stream_si256, which needs a 32-byte-aligned destination:

1#include <immintrin.h>
2
3// Fill n bytes without polluting the cache.
4// dst must be 32-byte aligned; n a multiple of 32.
5void fill_nt(void *dst, int value, size_t n) {
6    __m256i v = _mm256_set1_epi8((char)value);
7    for (size_t i = 0; i < n; i += 32)
8        _mm256_stream_si256((__m256i *)((char *)dst + i), v);
9    _mm_sfence();  // make the stores visible to other cores
10}

Using it safely

1. Align the destination to 32 bytes, or handle the unaligned head with regular stores.
2. Handle any tail shorter than 32 bytes with regular stores.
3. Issue _mm_sfence() before another thread reads the data.

4The instructions

x86 has a non-temporal store for each register width (Table 1).

5Where the lines cross

Putting the two regimes together gives the shape in Figure 2: regular stores are cheaper while the buffer fits in cache, and non-temporal stores win once it spills to DRAM. Exactly where the lines cross depends on the machine, which is what the next post measures.