Describe the bug
I have a Ubuntu 24.04 LTS VM with 2 vCPUs, and a Ubuntu 20.04 LTS VM with 2 vCPUs, both running on identical host hardware. If I compile, run, and benchmark redis on both VMs then redis runs much faster -- about 10% to 12% faster -- on Ubuntu 20 than Ubuntu 24. Why?
To reproduce
$ git clone https://github.com/redis/redis.git
$ cd redis/
$ make
$ make test
$ cd src
$ taskset --cpu-list 0 ./redis-server --port 9999
$ # Ubuntu 24 results:
$ export CONNECTIONS=50; taskset --cpu-list 1 ./redis-benchmark -h 127.0.0.1 -p 9999 -c $CONNECTIONS -q -t get -r 100000 -n 1000000 GET: 66414.29 requests per second, p50=0.367 msec
$ export CONNECTIONS=50; taskset --cpu-list 1 ./redis-benchmark -h 127.0.0.1 -p 9999 -c $CONNECTIONS -q -t get -r 100000 -n 1000000 GET: 67362.75 requests per second, p50=0.367 msec
$ export CONNECTIONS=50; taskset --cpu-list 1 ./redis-benchmark -h 127.0.0.1 -p 9999 -c $CONNECTIONS -q -t get -r 100000 -n 1000000 GET: 67838.00 requests per second, p50=0.359 msec
$ # Ubuntu 20 results:
$ export CONNECTIONS=50; taskset --cpu-list 1 ./redis-benchmark -h 127.0.0.1 -p 9999 -c $CONNECTIONS -q -t get -r 100000 -n 1000000 GET: 74968.14 requests per second, p50=0.327 msec
$ export CONNECTIONS=50; taskset --cpu-list 1 ./redis-benchmark -h 127.0.0.1 -p 9999 -c $CONNECTIONS -q -t get -r 100000 -n 1000000 GET: 75137.12 requests per second, p50=0.327 msec
$ export CONNECTIONS=50; taskset --cpu-list 1 ./redis-benchmark -h 127.0.0.1 -p 9999 -c $CONNECTIONS -q -t get -r 100000 -n 1000000 GET: 74432.45 requests per second, p50=0.327 msec
Expected behavior
I would expect performance to be similar, or?
Additional information
I got paranoid thinking that maybe all heavily TCP based servers are slower on Ubuntu 24, and so did a similar test with nginx, but nginx on Ubuntu 24 actually showed a 5% performance increase for Ubuntu 24 over Ubuntu 20.
Also, tried posting in this Ubuntu Community Discourse kernel discussion forum thread here [1], but no results for redis yet. Although there is a custom C program in the thread that works slower on Ubuntu 24.
[1] https://discourse.ubuntu.com/t/24-04-considerably-slower-than-20-04-or-22-04-for-some-high-system-percentage-usage-cases/41987/48
Comment From: simonhfot
Update: I've made some more discoveries about redis performance on Ubuntu 24. Namely, it only seems to be the localhost RPS performance which is borked. If I use redis-benchmark remotely, across the network, then redis on Ubuntu 24 actually performs better than Redis on Ubuntu 20. Below are the results of running redis-benchmark 3 times with 4 network configurations:
test redis-benchmark redis-server RPS1 RPS2 RPS3
==== =============== ============ ====== ====== ======
1 Ubuntu 20 Ubuntu 20 156543 152905 151285 <-- localhost
2 Ubuntu 24 Ubuntu 24 114916 116171 116360 <-- localhost
3 Ubuntu 24 Ubuntu 20 122549 123365 120845 <-- network
4 Ubuntu 20 Ubuntu 24 134553 129416 133280 <-- network
Comment From: sundb
@simonhfot thanks, i'll try to reproduce it.
Comment From: ShooterIT
redis-serve and memtier_benchmark are in the same VM.
bechmark command
memtier_benchmark --data-size 512 --ratio 1:1 --key-pattern P:P --key-minimum=1 --key-maximum 3000000 --distinct-client-seed --test-time 120 -c 50 -t 2 --hide-histogram
20.04
============================================================================================================================
Type Ops/sec Hits/sec Misses/sec Avg. Latency p50 Latency p99 Latency p99.9 Latency KB/sec
----------------------------------------------------------------------------------------------------------------------------
Sets 97691.81 --- --- 0.51288 0.47900 0.95900 1.11900 102331.15
Gets 97691.35 97691.35 0.00 0.51116 0.47900 0.95900 1.11100 101853.66
Waits 0.00 --- --- --- --- --- --- ---
Totals 195383.15 97691.35 0.00 0.51202 0.47900 0.95900 1.11100 204184.81
22.04
============================================================================================================================
Type Ops/sec Hits/sec Misses/sec Avg. Latency p50 Latency p99 Latency p99.9 Latency KB/sec
----------------------------------------------------------------------------------------------------------------------------
Sets 91087.06 --- --- 0.54961 0.51100 1.01500 1.39900 49780.21
Gets 91086.55 91086.55 0.00 0.54861 0.51100 1.01500 1.38300 49335.16
Waits 0.00 --- --- --- --- --- --- ---
Totals 182173.61 91086.55 0.00 0.54911 0.51100 1.01500 1.39100 99115.37
24.04
============================================================================================================================
Type Ops/sec Hits/sec Misses/sec Avg. Latency p50 Latency p99 Latency p99.9 Latency KB/sec
----------------------------------------------------------------------------------------------------------------------------
Sets 80164.23 --- --- 0.62438 0.59900 1.20700 1.77500 43810.70
Gets 80163.86 80163.86 0.00 0.62338 0.59900 1.20700 1.75900 43419.07
Waits 0.00 --- --- --- --- --- --- ---
Totals 160328.09 80163.86 0.00 0.62388 0.59900 1.20700 1.76700 87229.77
Comment From: ShooterIT
it seems higher version has worse performance, but i don't know the reason