When testing new LTS kernel 6.1, I noticed a big performance drop in redis.
Testing was done on a physical server with Intel Xeon CPU and in Qemu VM on the AMD EPYC CPU. Results are same.
Physical server:
test 5.10 6.1 %
PING_INLINE 67462 53818 79
PING_BULK 68282 52659 77
SET 68832 54942 79
GET 69204 54818 79
INCR 69803 54725 78
LPUSH 70407 54549 77
RPUSH 70368 54271 77
LPOP 70551 53940 76
RPOP 70343 54253 77
SADD 69676 53607 76
HSET 70313 54809 77
SPOP 69890 55227 79
ZADD 70541 54582 77
ZPOPMIN 69628 54911 78
LPUSH 70576 54010 76
LRANGE_100 46399 38692 83
LRANGE_300 21860 21076 96
LRANGE_500 15417 15011 97
LRANGE_600 12342 12475 101
MSET 70175 55129 78
average 81
Virtual server:
test 5.10 6.1 %
PING_INLINE 91751 72833 79
PING_BULK 89814 72727 80
SET 90195 75386 83
GET 88896 75843 85
INCR 91407 75278 82
LPUSH 89301 73838 82
RPUSH 90686 74183 81
LPOP 89381 73730 82
RPOP 89413 76254 85
SADD 89653 72859 81
HSET 90187 73540 81
SPOP 89269 72674 81
ZADD 88652 72296 81
ZPOPMIN 90212 72458 80
LPUSH 93222 73540 78
LRANGE_100 67272 55126 81
LRANGE_300 32142 29393 91
LRANGE_500 22112 21270 96
LRANGE_600 17748 17309 97
MSET 79365 77226 97
84
I did not find any difference in sysctl any other setting that can affect performance.
Did you know any setting that will take back performance from kernel 5.10?
Comment From: itamarhaber
/cc @filipecosta90
Comment From: filipecosta90
@vaclavskala we've detected certain variations ( positive and negative ) between kernel 5.15 and 6.0 using our benchmark specification (https://github.com/redis/redis-benchmarks-specification), but we are still investigating the root causes of both of them ( the improvements and regressions) with our partners -- this is being pushed by @slice4e and @ofekshenawa. We'll use this issue to discuss the findings as soon as we've bisected and fully confirmed their root cause.
Comment From: slice4e
@vaclavskala So far any regressions that i observed with kernel 6 were due to the benchmarks running at a lower CPU frequency. Still investigating the root cause... There were use cases in my testing where kernel 6 was better. @vaclavskala can you check the frequency while running your tests?
Comment From: vaclavskala
@vaclavskala So far any regressions that i observed with kernel 6 were due to the benchmarks running at a lower CPU frequency. Still investigating the root cause... There were use cases in my testing where kernel 6 was better. @vaclavskala can you check the frequency while running your tests?
Frequency was same. When testing on physical server, both servers had the same hw with the same CPU. And i checked that boost was enabled on both CPU.
When testing on VM, booth servers was on same proxmox node without any other VM. And when testing one server the other was idle.
Comment From: hailongeric
I can't reproduce this performance issue. I'm using Redis 7.0.4 and testing it on a virtual machine on Intel Xeon Gold 6133 CPU host. What version of Redis are you testing?