i got a problem, there is a small key less than 4KB, and a simple get command costs 1.6 seconds!
i've seen this pb on both redis 3.0.7 and 4.0.8,
as far as i know, time spent recorded in slowlog is ONLY time spent on executing command, so i do not understand how i could happen. could anyone help me on this case? thanks in advance.
PS: i only have total usage of CPU, but don't have usage of each core.
Comment From: charsyam
@romber2001 Could you check How much swap memory do you use? https://www.cyberciti.biz/faq/linux-which-process-is-using-swap/
and how about using redis-db mailing list? , I think it is more proper place to talk this kind of question :)
Comment From: romberli
@charsyam thanks for your reply,
here is the information of swap usage:
Comment From: romberli
@charsyam i'd love to use mailing list, but i have no idea about how to use it...
Comment From: charsyam
@romber2001 is it usually slow? or just Did you see it with "slowlog" command? and it is easy to use just send email to redis-db@googlegroups.com :)
Comment From: romberli
@charsyam usually, it's fast, but in some unkown circumstances, this simple command was recorded into slowlog, the slowlog threshold is 10ms.
Comment From: funny-falcon
Does any other software runs on the same host? Kernel may switch process for 10ms easily if there is almost full CPU utilization. In this case any command will magically became "slow".
Comment From: smartlee
@romber2001 have u sloved the issue,i'm encounter the same problem
Comment From: romberli
Does any other software runs on the same host? Kernel may switch process for 10ms easily if there is almost full CPU utilization. In this case any command will magically became "slow".
no, we use 6 dedicated virtual machine(4C/32G) to run redis only, we run 2 redis instances on a vm. the version is 4.0.8, and running in cluster mode with 12 instances(6 masters, 6 slaves)
Comment From: romberli
@romber2001 have u sloved the issue,i'm encounter the same problem
no, i'm still facing this pb, with no clue, the cpu usage is less than 20% on server, it's not a busy server, ops on each server is about ten or twenty thousands..
Comment From: smartlee
@romber2001 what's ur kernel and cpu model(lscpu output)
Comment From: romberli
hi @smartlee , sorry for late update, as i thought nobody pay attention to this issue... it's glad to get new information after such long time, here is my kernel and cpu information:
uname -a Linux 10-10-116-237 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 4 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 61 Model name: Intel Core Processor (Broadwell) Stepping: 2 CPU MHz: 2194.916 BogoMIPS: 4389.83 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K NUMA node0 CPU(s): 0-3
Comment From: smartlee
@romber2001 it's centos kernel ,cpu model is Broadwell ,not Skylake , so it's unrelated to my problem, but u can check tsc frequency to figure out whether it's constant or not
Comment From: charsyam
@smartlee what is your virtual machine manager? Do you use THP option? In some situation, fork system call takes much time.
Comment From: smartlee
@charsyam THP is disabled and i have found the root cause of this issue ,it's kernel bug
Comment From: romberli
@smartlee ,what kind of kernel bug? any document?
Comment From: oranagra
since this was resolved and not an issue with redis, i'll close it. p.s. another usual suspect for freezing processes may be numa_balancing
Comment From: smartlee
@romberli https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?h=v4.16-rc3&id=b511203093489eb1829cb4de86e8214752205ac6