we have upgrade our redis to 6.2.5 in production, info provided here, https://github.com/redis/redis/issues/9458 the io-threads config parameter is:
io-threads-do-reads yes
io-threads 6
and at a certain time the cpu will go high up to 300%+ cpu usage, it's a pubsub only redis instance , there's no extra log than normal log in the log file, and we've monitor the redis instance through python redis module by fetch the redis.info() data , and we found nothing abnormal there , no output surge or input surge , no instantaneous ops surge , no memory surge, but only cpu surge, i really don't know what happened , we only upgrade redis from 3.2.6 to 6.2.5 , added io-thread params , it is totally ok on redis 3.2.6 , is it the way we use io-thread in the pubsub scenario wrong ? or is it a bug on redis6 with pubsub ?
Comment From: filipecosta90
@xroa can you further describe your pubsub usage pattern ( number of consumers, producers, message size ) and provide further details if this is a reproducible pattern/does it happen immediately after start/if the CPU usage is constant or spiky and if there is the chance for you to collect some profile info for a deeper look at the issue?
Comment From: xroa
"PUBLISH" "queue:XX002007XXXX" "/0/1631275784/XX9996000022/98165416549816/XX00000000000/xxxxxx/xxx/10000000000\n{\"clientId\":\"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\",\"cmd\": \"mmm_call\", \"params\": {\"user\": 1999999999, \"tableId\": 1999999999888888, \"seatId\": 2, \"wpId\": 2116, \"fPosx\": 0.2633187472820282, \"fPosy\": 0.01666666381061077, \"fFlg\": 24, \"gamename\": 28, \"clientId\": \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\", \"aaaaaaa\": 3232323232323, \"action\": \"test\"}}"
The problem happens on the instance that we use as a message queue, just publish and subscribe , message size ranges from 50-40k , mostly the size is below 1k, size above 1k ocuppied only small percent, and the message format just like the one that i capture by monitor command above
Comment From: solemnpending
This problem cannot be reproduced using single-threaded mode...It is definitely related to enabling multithreading
Comment From: GuoZhaoran
I think it should have something to do with the CAS that the io thread does when the main thread is assigned to the io thread task, with the following code:
// the entry of io threads
void *IOThreadMain(void *myid) {
.......
while(1) {
/* Wait for start */
for (int j = 0; j < 1000000; j++) {
if (getIOPendingCount(id) != 0) break;
}
but there doesn't seem to be a more efficient way to communicate between threads than CAS!