Describe the bug

I have a Ubuntu 24.04 LTS VM with 2 vCPUs, and a Ubuntu 20.04 LTS VM with 2 vCPUs, both running on identical host hardware. If I compile, run, and benchmark redis on both VMs then redis runs much faster -- about 10% to 12% faster -- on Ubuntu 20 than Ubuntu 24. Why?

To reproduce

$ git clone https://github.com/redis/redis.git

$ cd redis/

$ make

$ make test

$ cd src

$ taskset --cpu-list 0 ./redis-server --port 9999

$ # Ubuntu 24 results:

$ export CONNECTIONS=50; taskset --cpu-list 1 ./redis-benchmark -h 127.0.0.1 -p 9999 -c $CONNECTIONS -q -t get -r 100000 -n 1000000 GET: 66414.29 requests per second, p50=0.367 msec

$ export CONNECTIONS=50; taskset --cpu-list 1 ./redis-benchmark -h 127.0.0.1 -p 9999 -c $CONNECTIONS -q -t get -r 100000 -n 1000000 GET: 67362.75 requests per second, p50=0.367 msec

$ export CONNECTIONS=50; taskset --cpu-list 1 ./redis-benchmark -h 127.0.0.1 -p 9999 -c $CONNECTIONS -q -t get -r 100000 -n 1000000 GET: 67838.00 requests per second, p50=0.359 msec

$ # Ubuntu 20 results:

$ export CONNECTIONS=50; taskset --cpu-list 1 ./redis-benchmark -h 127.0.0.1 -p 9999 -c $CONNECTIONS -q -t get -r 100000 -n 1000000 GET: 74968.14 requests per second, p50=0.327 msec

$ export CONNECTIONS=50; taskset --cpu-list 1 ./redis-benchmark -h 127.0.0.1 -p 9999 -c $CONNECTIONS -q -t get -r 100000 -n 1000000 GET: 75137.12 requests per second, p50=0.327 msec

$ export CONNECTIONS=50; taskset --cpu-list 1 ./redis-benchmark -h 127.0.0.1 -p 9999 -c $CONNECTIONS -q -t get -r 100000 -n 1000000 GET: 74432.45 requests per second, p50=0.327 msec

Expected behavior

I would expect performance to be similar, or?

Additional information

I got paranoid thinking that maybe all heavily TCP based servers are slower on Ubuntu 24, and so did a similar test with nginx, but nginx on Ubuntu 24 actually showed a 5% performance increase for Ubuntu 24 over Ubuntu 20.

Also, tried posting in this Ubuntu Community Discourse kernel discussion forum thread here [1], but no results for redis yet. Although there is a custom C program in the thread that works slower on Ubuntu 24.

[1] https://discourse.ubuntu.com/t/24-04-considerably-slower-than-20-04-or-22-04-for-some-high-system-percentage-usage-cases/41987/48

Comment From: simonhfot

Update: I've made some more discoveries about redis performance on Ubuntu 24. Namely, it only seems to be the localhost RPS performance which is borked. If I use redis-benchmark remotely, across the network, then redis on Ubuntu 24 actually performs better than Redis on Ubuntu 20. Below are the results of running redis-benchmark 3 times with 4 network configurations:

test  redis-benchmark  redis-server  RPS1    RPS2    RPS3
====  ===============  ============  ======  ======  ======
1     Ubuntu 20        Ubuntu 20     156543  152905  151285 <-- localhost
2     Ubuntu 24        Ubuntu 24     114916  116171  116360 <-- localhost

3     Ubuntu 24        Ubuntu 20     122549  123365  120845 <-- network
4     Ubuntu 20        Ubuntu 24     134553  129416  133280 <-- network

Comment From: sundb

@simonhfot thanks, i'll try to reproduce it.

Comment From: ShooterIT

redis-serve and memtier_benchmark are in the same VM.

bechmark command

memtier_benchmark --data-size 512 --ratio 1:1 --key-pattern P:P --key-minimum=1 --key-maximum 3000000  --distinct-client-seed --test-time 120 -c 50 -t 2  --hide-histogram

20.04

============================================================================================================================
Type         Ops/sec     Hits/sec   Misses/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec 
----------------------------------------------------------------------------------------------------------------------------
Sets        97691.81          ---          ---         0.51288         0.47900         0.95900         1.11900    102331.15 
Gets        97691.35     97691.35         0.00         0.51116         0.47900         0.95900         1.11100    101853.66 
Waits           0.00          ---          ---             ---             ---             ---             ---          --- 
Totals     195383.15     97691.35         0.00         0.51202         0.47900         0.95900         1.11100    204184.81 

22.04

============================================================================================================================
Type         Ops/sec     Hits/sec   Misses/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec 
----------------------------------------------------------------------------------------------------------------------------
Sets        91087.06          ---          ---         0.54961         0.51100         1.01500         1.39900     49780.21 
Gets        91086.55     91086.55         0.00         0.54861         0.51100         1.01500         1.38300     49335.16 
Waits           0.00          ---          ---             ---             ---             ---             ---          --- 
Totals     182173.61     91086.55         0.00         0.54911         0.51100         1.01500         1.39100     99115.37

24.04

============================================================================================================================
Type         Ops/sec     Hits/sec   Misses/sec    Avg. Latency     p50 Latency     p99 Latency   p99.9 Latency       KB/sec 
----------------------------------------------------------------------------------------------------------------------------
Sets        80164.23          ---          ---         0.62438         0.59900         1.20700         1.77500     43810.70 
Gets        80163.86     80163.86         0.00         0.62338         0.59900         1.20700         1.75900     43419.07 
Waits           0.00          ---          ---             ---             ---             ---             ---          --- 
Totals     160328.09     80163.86         0.00         0.62388         0.59900         1.20700         1.76700     87229.77 

Comment From: ShooterIT

it seems higher version has worse performance, but i don't know the reason