Describe the bug
in some cases, redis server just become freeze and unresponsive, failed to connect to redis server at all, and can't log in from redis insight at all. I don't know what exactly the issue, but based on timing, it could be one client disconnect from redis server, when the client is restarting itself. (FYI, the client is using Jedis library) any concern if the client is not shutdown graceful/properly. from the stack race below, seems like it might related to Search Indexing, while we created hundreds of temporary indexes from client using Jedis library, not sure if this raised any alarm.
To reproduce
don't have reproduce steps
Expected behavior
redis server should be responsive always
Additional information
stack trace when frozen:
#0 0x00007f8d352332ce in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
#1 0x00007f8d2c69956b in RedisSearchCtx_LockSpecWrite () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#2 0x00007f8d2c6ab45b in IndexSpec_DeleteDoc () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#3 0x00007f8d2c6abbfa in Indexes_DeleteMatchingWithSchemaRules () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#4 0x00007f8d2c681b16 in HashNotificationCallback () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#5 0x00000000004ff95f in moduleNotifyKeyspaceEvent.part.97.58377 (type=256, event=0x61abbf "expired", key=0x7f8c3d4ad620, dbid=0) at module.c:8724
#6 0x000000000050a445 in moduleNotifyKeyspaceEvent (dbid=0, key=0x7f8c3d4ad620, event=0x61abbf "expired", type=256) at notify.c:145
#7 notifyKeyspaceEvent (type=<optimized out>, event=<optimized out>, key=<optimized out>, dbid=<optimized out>, dbid=<optimized out>, key=<optimized out>, event=<optimized out>) at notify.c:114
#8 0x000000000050e6ea in deleteExpiredKeyAndPropagate (keyobj=0x7f8c3d4ad620, db=0x7f8d33531000) at db.c:1686
#9 expireIfNeeded.part.6.12916 (db=0x7f8d33531000, key=0x7f8c3d4ad620) at db.c:1813
#10 0x000000000050eb44 in expireIfNeeded (flags=<optimized out>, key=0x7f8c3d4ad620, db=0x7f8d33531000) at networking.c:4077
#11 lookupKey (db=0x7f8d33531000, key=0x7f8c3d4ad620, flags=0) at db.c:107
#12 0x0000000000516cdf in lookupKeyReadWithFlags (flags=0, key=0x7f8c3d4ad620, db=<optimized out>) at db.c:153
#13 RM_OpenKey (ctx=0x7ffcc4b3b790, keyname=0x7f8c3d4ad620, mode=1) at module.c:4005
#14 0x00007f8d2adad201 in redis_module::context::Context::open_key::h6e9944a47573023b ()
from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/rejson.so
#15 0x00007f8d2ad3a6bd in rejson::c_api::json_api_open_key_internal::heef0c3554720f6d6 ()
from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/rejson.so
#16 0x00007f8d2c6a1133 in getKeyCommonJSON () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#17 0x00007f8d2c69f8aa in loadIndividualKeys () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#18 0x00007f8d2c69e3b7 in rploaderNext () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#19 0x00007f8d2c640a5a in rpevalNext_project () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#20 0x00007f8d2c64abbd in Grouper_rpAccum () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#21 0x00007f8d2c69ec2e in rpsortNext_Accum () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#22 0x00007f8d2c638365 in sendChunk () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#23 0x00007f8d2c638bd7 in AREQ_Execute () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#24 0x00007f8d2c6396b3 in RSAggregateCommand () from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
#25 0x00000000004ffa9c in RedisModuleCommandDispatcher (c=0x7f8affc0b500) at module.c:917
#26 0x0000000000539808 in call (c=c@entry=0x7f8affc0b500, flags=flags@entry=3) at server.c:3519
#27 0x000000000053bb33 in processCommand (c=0x7f8affc0b500) at server.c:4160
#28 0x000000000057b7c7 in processCommandAndResetClient (c=0x7f8affc0b500) at networking.c:2466
#29 processInputBuffer (c=c@entry=0x7f8affc0b500) at networking.c:2574
#30 0x000000000057bf73 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#31 0x00000000004b1f55 in callHandler (handler=<optimized out>, conn=0x7f8a70b5df00) at connhelpers.h:79
#32 tlsHandleEvent (conn=0x7f8a70b5df00, mask=<optimized out>) at tls.c:720
#33 0x00000000004d5ab9 in aeProcessEvents (flags=27, eventLoop=0x7f8d3342a1e0) at ae.c:436
#34 aeMain (eventLoop=0x7f8d3342a1e0) at ae.c:496
#35 0x0000000000457210 in main (argc=<optimized out>, argv=0x7ffcc4b3bbc8) at server.c:7360
info register:
rax 0xfffffffffffffe00 -512
rbx 0x7ffcc4b3b100 140723608596736
rcx 0xffffffffffffffff -1
rdx 0x0 0
rsi 0x80 128
rdi 0x7f8d20b28dac 140244115688876
rbp 0x7f8d20b28c00 0x7f8d20b28c00
rsp 0x7ffcc4b3b0d8 0x7ffcc4b3b0d8
r8 0x7da 2010
r9 0x7f8d33400ff0 140244426952688
r10 0x0 0
r11 0x206 518
r12 0x7ffcc4b3b210 140723608597008
r13 0x7f8c3d4ad620 140240300463648
r14 0x7da 2010
r15 0x7ffcc4b3b100 140723608596736
rip 0x7f8d352332ce 0x7f8d352332ce <pthread_rwlock_wrlock+62>
eflags 0x206 [ PF IF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
redis server config:
maxmemory 300gb
maxmemory-policy allkeys-lru
appendonly no
save ""
timeout 0
io-threads 4
loglevel notice
logfile path
user admin on +@all ~* #somehash
user readOnly on +@read +info +ping +json.get +ts.range ~* #somehash
user readWrite on +@all -@dangerous +info +ping ~* #somehash
user default off
port 0
tls-port 6379
tls-cert-file path1
tls-key-file path2
tls-ca-cert-file path3
tls-auth-clients no
redis server log (notice): not any useful log around freezing time
38578:C 20 Aug 2024 12:47:08.767 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
38578:C 20 Aug 2024 12:47:08.768 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
38578:C 20 Aug 2024 12:47:08.768 * Redis version=7.2.1, bits=64, commit=00000000, modified=0, pid=38578, just started
38578:C 20 Aug 2024 12:47:08.769 * Configuration loaded
38578:M 20 Aug 2024 12:47:08.770 * monotonic clock: POSIX clock_gettime
38578:M 20 Aug 2024 12:47:08.772 * Running mode=standalone, port=6379.
38578:M 20 Aug 2024 12:47:08.773 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
38578:M 20 Aug 2024 12:47:08.777 * <search> Redis version found by RedisSearch : 7.2.1 - oss
38578:M 20 Aug 2024 12:47:08.778 * <search> RediSearch version 2.8.4 (Git=HEAD-7797d39)
38578:M 20 Aug 2024 12:47:08.778 * <search> Low level api version 1 initialized successfully
38578:M 20 Aug 2024 12:47:08.779 * <search> concurrent writes: OFF, gc: ON, prefix min length: 2, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, max number of search results: 10000, search pool size: 20, index pool size: 8,
38578:M 20 Aug 2024 12:47:08.780 * <search> Initialized thread pools!
38578:M 20 Aug 2024 12:47:08.780 * <search> Enabled role change notification
38578:M 20 Aug 2024 12:47:08.781 * Module 'search' loaded from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisearch.so
38578:M 20 Aug 2024 12:47:08.783 * <timeseries> RedisTimeSeries version 11006, git_sha=2a4739648bbe771fc60561b37fbc7f3ca4cc4f24
38578:M 20 Aug 2024 12:47:08.784 * <timeseries> Redis version found by RedisTimeSeries : 7.2.1 - oss
38578:M 20 Aug 2024 12:47:08.785 * <timeseries> loaded default CHUNK_SIZE_BYTES policy: 4096
38578:M 20 Aug 2024 12:47:08.786 * <timeseries> loaded server DUPLICATE_POLICY: block
38578:M 20 Aug 2024 12:47:08.786 * <timeseries> Setting default series ENCODING to: compressed
38578:M 20 Aug 2024 12:47:08.787 * <timeseries> Detected redis oss
38578:M 20 Aug 2024 12:47:08.788 * Module 'timeseries' loaded from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redistimeseries.so
38578:M 20 Aug 2024 12:47:08.790 * <ReJSON> Created new data type 'ReJSON-RL'
38578:M 20 Aug 2024 12:47:08.791 * <ReJSON> version: 20606 git sha: unknown branch: unknown
38578:M 20 Aug 2024 12:47:08.791 * <ReJSON> Exported RedisJSON_V1 API
38578:M 20 Aug 2024 12:47:08.792 * <ReJSON> Exported RedisJSON_V2 API
38578:M 20 Aug 2024 12:47:08.792 * <ReJSON> Exported RedisJSON_V3 API
38578:M 20 Aug 2024 12:47:08.793 * <ReJSON> Exported RedisJSON_V4 API
38578:M 20 Aug 2024 12:47:08.793 * <ReJSON> Enabled diskless replication
38578:M 20 Aug 2024 12:47:08.794 * Module 'ReJSON' loaded from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/rejson.so
38578:M 20 Aug 2024 12:47:08.794 * <search> Acquired RedisJSON_V4 API
38578:M 20 Aug 2024 12:47:08.796 * <bf> RedisBloom version 2.6.3 (Git=unknown)
38578:M 20 Aug 2024 12:47:08.797 * Module 'bf' loaded from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisbloom.so
38578:M 20 Aug 2024 12:47:08.799 * <redisgears_2> Created new data type 'GearsType'
38578:M 20 Aug 2024 12:47:08.800 * <redisgears_2> Detected redis oss
38578:M 20 Aug 2024 12:47:08.801 # <redisgears_2> could not initialize RedisAI_InitError
38578:M 20 Aug 2024 12:47:08.801 * <redisgears_2> Failed loading RedisAI API.
38578:M 20 Aug 2024 12:47:08.802 * <redisgears_2> RedisGears v2.0.12, sha='716eb57fcfc2d383ec0925a59563f1bf69e2caa3', build_type='release', built_for='Linux-rhel7.x86_64'.
38578:M 20 Aug 2024 12:47:08.805 * <redisgears_2> Registered backend: js.
38578:M 20 Aug 2024 12:47:08.806 * Module 'redisgears_2' loaded from /home/svc_cathedral/nomad/pxstlccath008/alloc/fa9f504b-64e7-d510-1b7c-f56885fb4dca/redis-task/local/redis-stack-server-7.2.0-v2/lib/redisgears.so
38578:M 20 Aug 2024 12:47:08.807 * The user 'default' is disabled (there is no 'on' modifier in the user description). Make sure this is not a configuration error.
38578:M 20 Aug 2024 12:47:08.819 * Server initialized
38578:M 20 Aug 2024 12:47:08.821 * Ready to accept connections tls
38578:M 20 Aug 2024 14:32:35.458 # Error accepting a client connection: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol (addr=127.0.0.1:54758 laddr=127.0.0.1:6379)
38578:M 20 Aug 2024 14:32:35.459 # Error accepting a client connection: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol (addr=10.49.30.56:37694 laddr=10.49.30.56:6379)
38578:M 20 Aug 2024 20:37:35.448 # Error accepting a client connection: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol (addr=127.0.0.1:34328 laddr=127.0.0.1:6379)
38578:M 20 Aug 2024 20:37:35.450 # Error accepting a client connection: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol (addr=10.49.30.56:45496 laddr=10.49.30.56:6379)
Comment From: sundb
@simon1990zcs thanks, can you move this issue to https://github.com/RediSearch/RediSearch/issues?q=sort%3Aupdated-desc+is%3Aissue+is%3Aopen?
Comment From: simon1990zcs
@simon1990zcs thanks, can you move this issue to https://github.com/RediSearch/RediSearch/issues?q=sort%3Aupdated-desc+is%3Aissue+is%3Aopen?
just created another one under RedisSearch, https://github.com/RediSearch/RediSearch/issues/4951, However, I am not 100% sure that's Search related though.