Getting the Cat Out of the Heap

2026-04-15T00:00:00+00:00
In performance critical areas like high-frequency trading arenas, latency is the difference between making or losing capital.</p>
In this blog, we will be optimizing a text book parser from processing ~440M ticks per second to processing ~25B ticks per second.</p>
Version 0: The Naive, Idiomatic C++ processor</h2>
A standard textbook feed handler consists of an order book having a generic Message</code> that is polymorphic. Hence, we create a base class, derive specific tick types and manage their lifetimes.</p>
#</span>pragma</span> once</span></span>
</span>
#</span>include</span> <</span>vector</span>></span></span>
#</span>include</span> <</span>memory</span>></span></span>
#</span>include</span> <</span>cstdint</span>></span></span>
</span>
// Generic market processing interface</span></span>
class</span> MarketMessage</span> {</span></span>
public</span>:</span></span>
    virtual</span> ~MarketMessage</span>()</span> = default;</span></span>
    virtual void</span> process</span>()</span> =</span> 0</span>;</span></span>
}</span>;</span></span>
</span>
// Specific implementation for adding an order</span></span>
class</span> AddOrderMessage</span> :</span> public</span> MarketMessage</span> {</span></span>
    uint64_t</span> timestamp</span>;</span></span>
    uint64_t</span> order_id</span>;</span></span>
    uint32_t</span> price</span>;</span></span>
    uint32_t</span> quantity</span>;</span></span>
    char</span> side</span>;</span> // Buy / Sell</span></span>
</span>
public</span>:</span></span>
    AddOrderMessage</span>(</span>uint64_t</span> ts</span>,</span> uint64_t</span> id</span>,</span> uint32_t</span> p</span>,</span> uint32_t</span> q</span>,</span> char</span> s</span>) :</span></span>
        timestamp</span>(</span>ts</span>),</span> order_id</span>(</span>id</span>),</span> price</span>(</span>p</span>),</span> quantity</span>(</span>q</span>),</span> side</span>(</span>s</span>) {}</span></span>
</span>
</span>
    void</span> process</span>()</span> override</span> {</span></span>
        // Dummy state machine update logic</span></span>
        asm volatile</span>("" : : "</span>g</span>"(</span>price</span>)</span>, </span>"</span>g</span>"(</span>quantity</span>) : "</span>memory</span>")</span>;</span></span>
    }</span></span>
}</span>;</span></span>
</span>
void</span> process_stream</span>(</span>const</span> std</span>::</span>vector</span><</span>std</span>::</span>shared_ptr</span><</span>MarketMessage</span>>></span>&</span> stream</span>) {</span></span>
    for</span> (</span>const auto&</span> msg </span>:</span> stream</span>) {</span></span>
        msg </span>-></span> process</span>()</span>;</span> // Latency trap</span></span>
    }</span></span>
}</span></span></code></pre>
Now, this code is clean, but it is fundamentally hostile to modern CPU architectures. The following Google Benchmark and perf</code> result will show it.</p>
--------------------------------------------------------------</span></span>
Benchmark</span>         Time         CPU  Iterations UserCounters...</span></span>
--------------------------------------------------------------</span></span>
Naive/10K</span>       8095</span> ns</span>     8067</span> ns</span>      64906</span> items_per_second=1.23966G/s</span></span>
Naive/100K</span>     80437</span> ns</span>    80079</span> ns</span>       8692</span> items_per_second=1.24877G/s</span></span>
Naive/1M</span>     2424241</span> ns</span>  2413241</span> ns</span>        290</span> items_per_second=414.381M/s</span></span>
</span>
 Performance</span> counter stats:</span></span>
</span>
       7,87,43,674</span>      cache-misses:u</span></span>
    5,35,97,05,947</span>      cache-references:u</span></span></code></pre>
We are hitting a cache-cliff</strong>. The CPU is spending more time waiting for main memory and looking up vtable addresses than it does calculating anything.</p>
Version 1: Data Oriented Design</h2>
To quote Martin Thompson, to achieve mechanical sympathy, we must stop thinking about 'Objects' and start thinking about 'Data'. The CPU prefetcher loves predictable, contiguous memory. We need to kill the heap allocations, remove the vtables and pack our data tightly aligning with the 64 byte L1 cache lines.</p>
enum class</span> MsgType</span> :</span> uint8_t</span> {</span></span>
    ADD</span>,</span></span>
    CANCEL</span>,</span></span>
    MODIFY</span></span>
}</span>;</span></span>
</span>
// Flat contiguous POD struct</span></span>
struct</span> MarketMessagePod</span> {</span></span>
    uint64_t</span> timestamp</span>;</span></span>
    uint64_t</span> order_id</span>;</span></span>
    uint32_t</span> price</span>;</span></span>
    uint32_t</span> quantity</span>;</span></span>
    MsgType type</span>;</span></span>
    char</span> side</span>;</span></span>
}</span>;</span></span>
</span>
void</span> process_stream</span>(</span>const</span> std</span>::</span>vector</span><</span>MarketMessagePod</span>></span>&</span> stream</span>) {</span></span>
    for</span> (</span>const auto&</span> msg </span>:</span> stream</span>) {</span></span>
        // Switch-case replaces virtual dispatch</span></span>
        switch</span> (</span>msg</span>.</span>type</span>) {</span></span>
            case</span> MsgType</span>::</span>ADD</span>:</span></span>
                // Dummy state machine update</span></span>
                asm volatile</span>("" : : "</span>g</span>"(</span>msg</span>.</span>price</span>)</span>, </span>"</span>g</span>"(</span>msg</span>.</span>quantity</span>) : "</span>memory</span>")</span>;</span></span>
                break;</span></span>
            // ...</span></span>
        }</span></span>
    }</span></span>
}</span></span></code></pre>
By switching to an array of structs (AoS), the results are immediate:</p>
------------------------------------------------------------</span></span>
Benchmark</span>       Time         CPU  Iterations UserCounters...</span></span>
------------------------------------------------------------</span></span>
DoD/10K</span>      2760</span> ns</span>     2752</span> ns</span>      253335</span> items_per_second=3.63315G/s</span></span>
DoD/100K</span>    31076</span> ns</span>    30974</span> ns</span>       22501</span> items_per_second=3.22853G/s</span></span>
DoD/1M</span>    1074340</span> ns</span>  1068376</span> ns</span>         650</span> items_per_second=936M/s</span></span>
</span>
 Performance</span> counter stats:</span></span>
</span>
       9,94,29,596</span>      cache-misses:u</span>                                                        </span></span>
    7,49,13,99,026</span>      cache-references:u</span>                                                    </span></span></code></pre>
We can now process double the ticks at ~936M per second. But, this increased the cache misses but also increased the number of cache references as well. We can find the percentage of cache misses as:
$$
\frac{cache\_misses}{cache\_references} \times 100 \%
$$</p>
Using this formula, the cache-miss rate for the initial (version 1) approach is $1.46 \%$. For the new data oriented design, this rate becomes $1.327 \%$. Hence, the cache-miss rate decreases as well.</p>
Version 2: Static Resolution, CRTP and Branch Hinting</h2>
Now, the memory is aligned, but now we face a control-flow bottleneck. The switch</code> statement introduces conditional branches, risking pipeline flushes. We need to use Curiously Recurring Template Pattern (CRTP) and compiler attributes to eliminate branching entirely. We inherit the MarketMessagePod</code> from the V2 data oriented design.</p>
#</span>include</span> <</span>data_oriented.hpp</span>></span></span>
</span>
template</span> <</span>typename</span> Derived</span>></span></span>
class</span> MessageHandler</span> {</span></span>
public</span>:</span></span>
    inline void</span> handle</span>(</span>const</span> MarketMessagePod</span>&</span> msg</span>) {</span></span>
        static_cast<</span>Derived</span>*></span>(</span>this</span>) -></span> process_impl</span>(</span>msg</span>)</span>;</span></span>
    }</span></span>
}</span>;</span></span>
</span>
class</span> FastOrderBook</span> :</span> public</span> MessageHandler</span><</span>FastOrderBook</span>> {</span></span>
public</span>:</span></span>
    inline void</span> process_impl</span>(</span>const</span> MarketMessagePod</span>&</span> msg</span>) {</span></span>
        if</span> (</span>msg</span>.</span>type</span> ==</span> MsgType</span>::</span>ADD</span>) [[</span>likely</span>]] {</span></span>
            asm volatile</span>("" : : "</span>g</span>"(</span>msg</span>.</span>price</span>)</span>, </span>"</span>g</span>"(</span>msg</span>.</span>quantity</span>) : "</span>memory</span>")</span>;</span></span>
        }</span> </span></span>
        else</span> [[</span>unlikely</span>]] {</span></span>
            asm volatile</span>("" : : "</span>g</span>"(</span>msg</span>.</span>order_id</span>) : "</span>memory</span>")</span>;</span></span>
        }</span></span>
    }</span></span>
}</span>;</span></span>
</span>
void</span> process_stream</span>(</span>const</span> std</span>::</span>vector</span><</span>MarketMessagePod</span>></span>&</span> stream</span>,</span> FastOrderBook</span>&</span> book</span>) {</span></span>
    const size_t</span> size </span>=</span> stream</span>.</span>size</span>()</span>;</span></span>
    const</span> MarketMessagePod</span>*</span> __restrict__ data </span>=</span> stream</span>.</span>data</span>()</span>;</span></span>
</span>
    #</span>pragma</span> GCC unroll</span> 4</span></span>
    for</span> (</span>size_t</span> i </span>=</span> 0</span>;</span> i </span><</span> size</span>;</span> i</span>++</span>) {</span></span>
        book</span>.</span>handle</span>(</span>data</span>[</span>i</span>])</span>;</span></span>
    }</span></span>
}</span></span></code></pre>
The throughput now jumps to 1.15 billion per ticks.</p>
------------------------------------------------------------------</span></span>
Benchmark</span>             Time        CPU   Iterations UserCounters...</span></span>
------------------------------------------------------------------</span></span>
SR_CRTP_BH/10K</span>     3206</span> ns</span>    3198</span> ns</span>       185542</span> items_per_second=3.12661G/s</span></span>
SR_CRTP_BH/100K</span>   32419</span> ns</span>   32318</span> ns</span>        21525</span> items_per_second=3.09423G/s</span></span>
SR_CRTP_BH/1M</span>    874833</span> ns</span>  869572</span> ns</span>          795</span> items_per_second=1.14999G/s</span></span>
</span>
 Performance</span> counter stats for:</span></span>
</span>
       8,59,02,211</span>      cache-misses:u</span>                                                        </span></span>
    6,40,36,81,500</span>      cache-references:u</span></span></code></pre>Version 3: Silicon Symphony: SIMD Vectorization</h2>
Writing custom deep learning libraries teaches you very quickly that calculating token generation rates relies entirely on packing the CPU's wide registers. Processing scalar values is a waste of silicon. We can apply that exact same SIMD acceleration to our market data parser.</p>
To use AVX2 intrinsics, we must flip our architecture inside out to a Struct of Arrays (SoA) to provide the 256-bit registers with uniform data.</p>
#</span>include</span> <</span>immintrin.h</span>></span> // AVX/SSE Intrinsics</span></span>
</span>
struct</span> MarketDataSoA</span> {</span></span>
    alignas</span>(</span>64</span>)</span> std</span>::</span>vector</span><uint64_t></span> timestamps</span>;</span></span>
    alignas</span>(</span>64</span>)</span> std</span>::</span>vector</span><uint64_t></span> order_ids</span>;</span></span>
    alignas</span>(</span>64</span>)</span> std</span>::</span>vector</span><uint32_t></span> prices</span>;</span></span>
    alignas</span>(</span>64</span>)</span> std</span>::</span>vector</span><uint32_t></span> quantities</span>;</span></span>
    alignas</span>(</span>64</span>)</span> std</span>::</span>vector</span><uint8_t></span> types</span>;</span> </span></span>
</span>
    void</span> reserve</span>(</span>size_t</span> size</span>) {</span></span>
        timestamps</span>.</span>reserve</span>(</span>size</span>)</span>;</span> order_ids</span>.</span>reserve</span>(</span>size</span>)</span>;</span></span>
        prices</span>.</span>reserve</span>(</span>size</span>)</span>;</span> quantities</span>.</span>reserve</span>(</span>size</span>)</span>;</span> types</span>.</span>reserve</span>(</span>size</span>)</span>;</span></span>
    }</span></span>
}</span>;</span></span>
</span>
inline void</span> process_stream_simd</span>(</span>const</span> MarketDataSoA</span>&</span> book</span>,</span> size_t</span> size</span>) {</span></span>
    const uint32_t*</span> __restrict__ qty_ptr </span>=</span> book</span>.</span>quantities</span>.</span>data</span>()</span>;</span></span>
    size_t</span> i </span>=</span> 0</span>;</span></span>
    </span></span>
    __m256i vector_accumulator </span>=</span> _mm256_setzero_si256</span>()</span>;</span></span>
</span>
    for</span> (</span>;</span> i </span>+</span> 8</span> <=</span> size</span>;</span> i </span>+=</span> 8</span>) {</span></span>
        __m256i q_vec </span>=</span> _mm256_loadu_si256</span>(</span>reinterpret_cast<const</span> __m256i</span>*></span>(</span>qty_ptr </span>+</span> i</span>))</span>;</span></span>
        vector_accumulator </span>=</span> _mm256_add_epi32</span>(</span>vector_accumulator</span>,</span> q_vec</span>)</span>;</span></span>
    }</span></span>
</span>
    for</span> (</span>;</span> i </span><</span> size</span>;</span> i</span>++</span>) {</span></span>
        asm volatile</span>("" : : "</span>g</span>"(</span>qty_ptr</span>[</span>i</span>]) : "</span>memory</span>")</span>;</span></span>
    }</span></span>
    </span></span>
    asm volatile</span>("" : : "</span>x</span>"(</span>vector_accumulator</span>) : "</span>memory</span>")</span>;</span></span>
}</span></span></code></pre>------------------------------------------------------------</span></span>
Benchmark</span>       Time        CPU   Iterations UserCounters...</span></span>
------------------------------------------------------------</span></span>
SIMD/10K</span>      298</span> ns</span>     297</span> ns</span>      2340635</span> items_per_second=33.6261G/s</span></span>
SIMD/100K</span>    3374</span> ns</span>    3364</span> ns</span>       207721</span> items_per_second=29.7253G/s</span></span>
SIMD/1M</span>     38100</span> ns</span>   37976</span> ns</span>        18422</span> items_per_second=26.3322G/s</span></span>
</span>
 Performance</span> counter stats for:</span></span>
</span>
      18,69,98,149</span>      cache-misses:u</span>                                                        </span></span>
    9,39,03,07,057</span>      cache-references:u</span></span></code></pre>
26 Billion Ticks per second</strong>. That is a staggering 60x gain in performance. This is still not enough for low latency trading.</p>
Version 4: Kernel Bypass</h2>
A single Linux scheduler tick takes about 3 to 5 microseconds. In that tiny window, our Version 3 SIMD loop could have processed over 100,000 messages. If the OS decides to pause our thread, we lose the trade.</p>
We must guarantee determinism by isolating our thread. A custom kernel supports tuning the bootloader parameters (isolcpus=2 nohz_full=2 rcu_nocbs=2</code>) to completely evict the OS from a specific core.</p>
Then, we bind our execution strictly to that physical silicon:</p>
#</span>pragma</span> once</span></span>
#</span>include</span> <</span>pthread.h</span>></span></span>
#</span>include</span> <</span>sched.h</span>></span></span>
#</span>include</span> <</span>stdexcept</span>></span></span>
#</span>include</span> <</span>iostream</span>></span></span>
#</span>include</span> <</span>simd_vectorized.hpp</span>></span> // Reusing our peak SIMD logic</span></span>
</span>
class</span> HardwareIsolator</span> {</span></span>
public</span>:</span></span>
    static void</span> pin_thread_to_core</span>(</span>int</span> core_id</span>) {</span></span>
        cpu_set_t</span> cpuset</span>;</span></span>
        CPU_ZERO</span>(</span>&</span>cpuset</span>)</span>;</span></span>
        CPU_SET</span>(</span>core_id</span>,</span> &</span>cpuset</span>)</span>;</span></span>
        </span></span>
        pthread_t</span> current_thread </span>=</span> pthread_self</span>()</span>;</span></span>
        if</span> (</span>pthread_setaffinity_np</span>(</span>current_thread</span>,</span> sizeof</span>(</span>cpu_set_t</span>),</span> &</span>cpuset</span>)</span> !=</span> 0</span>) {</span></span>
            std</span>::</span>cerr </span><<</span> "</span>WARNING: Failed to pin thread to core </span>"</span> <<</span> core_id </span><<</span> "</span>\n</span>"</span>;</span></span>
        }</span></span>
    }</span></span>
}</span>;</span></span>
</span>
// The V4 processing function simply wraps V3 but ensures we are pinned</span></span>
inline void</span> process_stream</span>(</span>const</span> MarketDataSoA</span>&</span> book</span>,</span> size_t</span> size</span>,</span> int</span> target_core</span>) {</span></span>
    HardwareIsolator</span>::</span>pin_thread_to_core</span>(</span>target_core</span>)</span>;</span></span>
    </span></span>
    // Execute the hyper-optimized SIMD loop</span></span>
    process_stream_simd</span>(</span>book</span>,</span> size</span>)</span>;</span></span>
}</span></span></code></pre>
Running the benchmark with FIFO real time priority (with sudo chrt -f 99</code>), we get:</p>
----------------------------------------------------------</span></span>
Benchmark</span>       Time       CPU  Iterations UserCounters...</span></span>
----------------------------------------------------------</span></span>
Metal/10K</span>    1222</span> ns</span>   1219</span> ns</span>      578139</span> items_per_second=8.20443G/s</span></span>
Metal/100K</span>   4604</span> ns</span>   4592</span> ns</span>      142760</span> items_per_second=21.7761G/s</span></span>
Metal/1M</span>    39963</span> ns</span>  39890</span> ns</span>       17527</span> items_per_second=25.0692G/s</span></span>
</span>
 Performance</span> counter stats for:</span></span>
</span>
      12,87,16,735</span>      cache-misses:u</span>                                                        </span></span>
    6,94,77,45,302</span>      cache-references:u</span></span></code></pre>
Bypassing the OS slightly decreased our peak throughput from ~26 G/s to ~25 G/s. This is because we have hit the physical limits of the silicon's memory bandwidth.</p>
But in low latency arenas, peak throughput is a vanity metric; jitter is the only metric that pays. We traded a tiny fraction of theoretical throughput for absolute, unwavering determinism.</p>
The Numbers</h2>
Method</th> Cache Misses</th> Cache References</th> Throughput</th></tr></thead>

Naive</td> 78743674</td> 5359705947</td> 414.381M/s</td></tr>
Data Oriented Design</td> 99429596</td> 7491399026</td> 936M/s</td></tr>
Static Processing + CRTP</td> 85902211</td> 6403681500</td> 1.15G/s</td></tr>
SIMD Processing</td> 186998149</td> 9390307057</td> 26.3322G/s</td></tr>
Metal</td> 128716735</td> 6947745302</td> 25.0692G/s</td></tr>
</tbody></table>
Source</h2>

NOTE: The test was done on a Ryzen 7 7435HS with sudo cpupower frequency-set -g performance</code></p>
</blockquote>
The code files for this implementation are available here</a> and are free to use under the Boost Software License 1.0</a>.</p>
Siddhesh Badnapurkar

Getting the Cat Out of the Heap

Source</h2>

NOTE: The test was done on a Ryzen 7 7435HS with `sudo cpupower frequency-set -g performance</code></p> </blockquote>`
`The code files for this implementation are available here</a> and are free to use under the Boost Software License 1.0</a>.</p>`

Method</th>	Cache Misses</th>	Cache References</th>	Throughput</th></tr></thead>
Naive</td>	78743674</td>	5359705947</td>	414.381M/s</td></tr>
Data Oriented Design</td>	99429596</td>	7491399026</td>	936M/s</td></tr>
Static Processing + CRTP</td>	85902211</td>	6403681500</td>	1.15G/s</td></tr>
SIMD Processing</td>	186998149</td>	9390307057</td>	26.3322G/s</td></tr>
Metal</td>	128716735</td>	6947745302</td>	25.0692G/s</td></tr> </tbody></table> Source</h2> NOTE: The test was done on a Ryzen 7 7435HS with `sudo cpupower frequency-set -g performance</code></p> </blockquote>` `The code files for this implementation are available here</a> and are free to use under the Boost Software License 1.0</a>.</p>`
Siddhesh Badnapurkar

Getting the Cat Out of the Heap

Source</h2> NOTE: The test was done on a Ryzen 7 7435HS with sudo cpupower frequency-set -g performance</code></p> </blockquote> The code files for this implementation are available here</a> and are free to use under the Boost Software License 1.0</a>.</p>

Source</h2>

NOTE: The test was done on a Ryzen 7 7435HS with `sudo cpupower frequency-set -g performance</code></p> </blockquote>`
`The code files for this implementation are available here</a> and are free to use under the Boost Software License 1.0</a>.</p>`
