Redis [CRASH] redis 6.2.6 on centos7 - out of memory allocating 120kb

Crash report

7369:M 22 Feb 2022 01:02:09.012 # Out Of Memory allocating 122916 bytes!

=== REDIS BUG REPORT START: Cut & paste starting from here ===
7369:M 22 Feb 2022 01:02:09.012 # ------------------------------------------------
7369:M 22 Feb 2022 01:02:09.012 # !!! Software Failure. Press left mouse button to continue
7369:M 22 Feb 2022 01:02:09.012 # Guru Meditation: Redis aborting for OUT OF MEMORY. Allocating 122916 bytes! #server.c:5968

------ STACK TRACE ------

Backtrace:
/usr/bin/redis-server *:6379(redisOutOfMemoryHandler+0x31)[0x55e7c1210411]
/usr/bin/redis-server *:6379(zmalloc_usable+0x2e)[0x55e7c121d39e]
/usr/bin/redis-server *:6379(sdsMakeRoomFor+0xd0)[0x55e7c121a780]
/usr/bin/redis-server *:6379(processMultibulkBuffer+0x4c8)[0x55e7c122b478]
/usr/bin/redis-server *:6379(processInputBuffer+0x1c2)[0x55e7c122ba02]
/usr/bin/redis-server *:6379(+0xfb858)[0x55e7c12c1858]
/usr/bin/redis-server *:6379(aeProcessEvents+0x2d9)[0x55e7c120c319]
/usr/bin/redis-server *:6379(aeMain+0x1d)[0x55e7c120c5ad]
/usr/bin/redis-server *:6379(main+0x330)[0x55e7c1208d80]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fdccfcc5555]
/usr/bin/redis-server *:6379(+0x432c1)[0x55e7c12092c1]

------ INFO OUTPUT ------
# Server
redis_version:6.2.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:4ab9a06393930489
redis_mode:standalone
os:Linux 3.10.0-1160.53.1.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:8.3.1
process_id:7369
process_supervised:systemd
run_id:7c4e948a55557cd0fd8ab9331b29d3efff1d243e
tcp_port:6379
server_time_usec:1645480929012737
uptime_in_seconds:24134
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:1313760
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0

# Clients
connected_clients:135
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:392
client_recent_max_output_buffer:49176
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:1923333352
used_memory_human:1.79G
used_memory_rss:2097643520
used_memory_rss_human:1.95G
used_memory_peak:2039130584
used_memory_peak_human:1.90G
used_memory_peak_perc:94.32%
used_memory_overhead:30357776
used_memory_startup:811800
used_memory_dataset:1892975576
used_memory_dataset_perc:98.46%
allocator_allocated:1923217400
allocator_active:2057211904
allocator_resident:2099306496
total_system_memory:16826032128
total_system_memory_human:15.67G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:4294967296
maxmemory_human:4.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.07
allocator_frag_bytes:133994504
allocator_rss_ratio:1.02
allocator_rss_bytes:42094592
rss_overhead_ratio:1.00
rss_overhead_bytes:-1662976
mem_fragmentation_ratio:1.09
mem_fragmentation_bytes:175156824
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:1251560
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:34373095
rdb_bgsave_in_progress:0
rdb_last_save_time:1645456795
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:3720116
total_commands_processed:32133553
instantaneous_ops_per_sec:1591
total_net_input_bytes:21560338157
total_net_output_bytes:140226842073
instantaneous_input_kbps:1268.64
instantaneous_output_kbps:5638.57
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:4168566
expired_stale_perc:23.56
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:27999
evicted_keys:0
keyspace_hits:51455189
keyspace_misses:4128352
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:36632667
total_writes_processed:32542844
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:9cec2154a2da224ffd7e7383ffd0dd0372133991
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:925.125467
used_cpu_user:816.302013
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:923.629905
used_cpu_user_main_thread:815.790350

# Modules

# Commandstats
cmdstat_get:calls=8710012,usec=56798695,usec_per_call=6.52,rejected_calls=0,failed_calls=0
cmdstat_exec:calls=1217,usec=42938,usec_per_call=35.28,rejected_calls=0,failed_calls=0
cmdstat_hgetall:calls=2011290,usec=66486440,usec_per_call=33.06,rejected_calls=0,failed_calls=0
cmdstat_multi:calls=1217,usec=499,usec_per_call=0.41,rejected_calls=0,failed_calls=0
cmdstat_info:calls=115,usec=8256,usec_per_call=71.79,rejected_calls=0,failed_calls=0
cmdstat_monitor:calls=1,usec=1,usec_per_call=1.00,rejected_calls=0,failed_calls=0
cmdstat_exists:calls=2664716,usec=7788698,usec_per_call=2.92,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=804,usec=430,usec_per_call=0.53,rejected_calls=0,failed_calls=0
cmdstat_expire:calls=3217891,usec=5863179,usec_per_call=1.82,rejected_calls=0,failed_calls=0
cmdstat_select:calls=3718390,usec=3505039,usec_per_call=0.94,rejected_calls=0,failed_calls=0
cmdstat_set:calls=2563286,usec=7748858,usec_per_call=3.02,rejected_calls=0,failed_calls=0
cmdstat_setex:calls=7694399,usec=35068244,usec_per_call=4.56,rejected_calls=0,failed_calls=0
cmdstat_keys:calls=3,usec=128265,usec_per_call=42755.00,rejected_calls=0,failed_calls=0
cmdstat_command:calls=1,usec=1350,usec_per_call=1350.00,rejected_calls=0,failed_calls=0
cmdstat_hmset:calls=665386,usec=27697658,usec_per_call=41.63,rejected_calls=0,failed_calls=0
cmdstat_mget:calls=883230,usec=87395296,usec_per_call=98.95,rejected_calls=0,failed_calls=0
cmdstat_del:calls=1595,usec=6266,usec_per_call=3.93,rejected_calls=0,failed_calls=0

# Errorstats

# Cluster
cluster_enabled:0

# Keyspace
db11:keys=311104,expires=310902,avg_ttl=1593885

------ CLIENT LIST OUTPUT ------
id=3720075 addr=10.0.3.83:51980 laddr=10.0.3.74:6379 fd=124 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720077 addr=10.0.3.71:42694 laddr=10.0.3.74:6379 fd=126 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720079 addr=10.0.3.86:20114 laddr=10.0.3.74:6379 fd=129 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3720081 addr=10.0.3.43:29988 laddr=10.0.3.74:6379 fd=130 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720083 addr=10.0.3.87:35382 laddr=10.0.3.74:6379 fd=132 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720084 addr=10.0.3.71:42698 laddr=10.0.3.74:6379 fd=14 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20872 events=r cmd=mget user=default redir=-1
id=3720085 addr=10.0.3.81:59952 laddr=10.0.3.74:6379 fd=92 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3720086 addr=10.0.3.71:42706 laddr=10.0.3.74:6379 fd=96 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720047 addr=10.0.3.73:32574 laddr=10.0.3.74:6379 fd=48 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=exists user=default redir=-1
id=3720049 addr=10.0.3.73:32576 laddr=10.0.3.74:6379 fd=53 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3720055 addr=10.0.3.81:59940 laddr=10.0.3.74:6379 fd=69 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20872 events=r cmd=mget user=default redir=-1
id=3719993 addr=10.0.3.71:42668 laddr=10.0.3.74:6379 fd=113 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3720009 addr=10.0.3.84:39570 laddr=10.0.3.74:6379 fd=110 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=exists user=default redir=-1
id=3720011 addr=10.0.3.84:39572 laddr=10.0.3.74:6379 fd=12 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3720012 addr=10.0.3.43:29936 laddr=10.0.3.74:6379 fd=50 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3720001 addr=10.0.3.29:53442 laddr=10.0.3.74:6379 fd=19 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=exists user=default redir=-1
id=3720003 addr=10.0.3.71:42672 laddr=10.0.3.74:6379 fd=85 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=exists user=default redir=-1
id=3719970 addr=10.0.3.43:29908 laddr=10.0.3.74:6379 fd=91 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3719977 addr=10.0.3.43:29920 laddr=10.0.3.74:6379 fd=61 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hgetall user=default redir=-1
id=3719949 addr=10.0.3.82:59322 laddr=10.0.3.74:6379 fd=72 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719954 addr=10.0.3.29:53392 laddr=10.0.3.74:6379 fd=68 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3719956 addr=10.0.3.82:59326 laddr=10.0.3.74:6379 fd=76 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=exists user=default redir=-1
id=3719929 addr=10.0.3.67:31902 laddr=10.0.3.74:6379 fd=70 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=exists user=default redir=-1
id=3719934 addr=10.0.3.29:53368 laddr=10.0.3.74:6379 fd=30 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719935 addr=10.0.3.70:43178 laddr=10.0.3.74:6379 fd=25 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3719936 addr=10.0.3.66:35490 laddr=10.0.3.74:6379 fd=31 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hgetall user=default redir=-1
id=3719900 addr=10.0.3.29:53360 laddr=10.0.3.74:6379 fd=93 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=expire user=default redir=-1
id=3719901 addr=10.0.3.67:31878 laddr=10.0.3.74:6379 fd=101 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61480 events=r cmd=setex user=default redir=-1
id=3719905 addr=10.0.3.81:59868 laddr=10.0.3.74:6379 fd=114 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719883 addr=10.0.3.71:42594 laddr=10.0.3.74:6379 fd=39 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61480 events=r cmd=setex user=default redir=-1
id=3719887 addr=10.0.3.85:53806 laddr=10.0.3.74:6379 fd=95 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719850 addr=10.0.3.43:29828 laddr=10.0.3.74:6379 fd=32 name= age=2 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719860 addr=10.0.3.29:53340 laddr=10.0.3.74:6379 fd=35 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719825 addr=10.0.3.71:42562 laddr=10.0.3.74:6379 fd=52 name= age=2 idle=2 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hgetall user=default redir=-1
id=3719826 addr=10.0.3.69:46462 laddr=10.0.3.74:6379 fd=56 name= age=2 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719807 addr=10.0.3.69:46450 laddr=10.0.3.74:6379 fd=28 name= age=2 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719782 addr=10.0.3.71:42544 laddr=10.0.3.74:6379 fd=15 name= age=2 idle=2 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hgetall user=default redir=-1
id=3719758 addr=10.0.3.73:32452 laddr=10.0.3.74:6379 fd=109 name= age=2 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=expire user=default redir=-1
id=3719711 addr=10.0.3.29:53238 laddr=10.0.3.74:6379 fd=78 name= age=2 idle=2 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=exists user=default redir=-1
id=3719693 addr=10.0.3.84:39386 laddr=10.0.3.74:6379 fd=106 name= age=2 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719626 addr=10.0.3.43:29700 laddr=10.0.3.74:6379 fd=67 name= age=2 idle=2 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719530 addr=10.0.3.69:46232 laddr=10.0.3.74:6379 fd=23 name= age=3 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719532 addr=10.0.3.69:46234 laddr=10.0.3.74:6379 fd=84 name= age=3 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61480 events=r cmd=setex user=default redir=-1
id=3719463 addr=10.0.3.70:42828 laddr=10.0.3.74:6379 fd=82 name= age=3 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61480 events=r cmd=setex user=default redir=-1
id=3695106 addr=10.0.3.88:55272 laddr=10.0.3.74:6379 fd=140 name= age=115 idle=115 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=expire user=default redir=-1
id=3692130 addr=10.0.3.72:22916 laddr=10.0.3.74:6379 fd=90 name= age=127 idle=0 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=65530 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=86040 events=r cmd=expire user=default redir=-1
id=3464340 addr=10.0.3.88:42472 laddr=10.0.3.74:6379 fd=44 name= age=1086 idle=1086 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3410895 addr=10.0.3.88:25512 laddr=10.0.3.74:6379 fd=38 name= age=1309 idle=10 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3360731 addr=10.0.3.88:12642 laddr=10.0.3.74:6379 fd=115 name= age=1534 idle=1534 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=expire user=default redir=-1
id=3345707 addr=10.0.3.88:63818 laddr=10.0.3.74:6379 fd=62 name= age=1606 idle=1606 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3266484 addr=10.0.3.88:39686 laddr=10.0.3.74:6379 fd=42 name= age=2015 idle=2015 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3266486 addr=10.0.3.88:39688 laddr=10.0.3.74:6379 fd=7 name= age=2015 idle=2015 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3266487 addr=10.0.3.88:39690 laddr=10.0.3.74:6379 fd=65 name= age=2015 idle=2015 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=1154017 addr=10.0.3.88:29694 laddr=10.0.3.74:6379 fd=13 name= age=18115 idle=9057 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=747638 addr=10.0.3.88:27268 laddr=10.0.3.74:6379 fd=27 name= age=19916 idle=15251 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=3276391 addr=10.0.3.88:42856 laddr=10.0.3.74:6379 fd=80 name= age=1962 idle=1058 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=expire user=default redir=-1
id=3277300 addr=10.0.3.88:43214 laddr=10.0.3.74:6379 fd=86 name= age=1957 idle=1957 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=1901207 addr=10.0.3.88:46132 laddr=10.0.3.74:6379 fd=58 name= age=13492 idle=12380 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=2051450 addr=10.0.3.88:60576 laddr=10.0.3.74:6379 fd=59 name= age=12387 idle=9242 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=724613 addr=10.0.3.88:16326 laddr=10.0.3.74:6379 fd=37 name= age=20090 idle=3304 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=889577 addr=10.0.3.88:11782 laddr=10.0.3.74:6379 fd=34 name= age=19282 idle=6628 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=3588733 addr=10.0.3.88:22644 laddr=10.0.3.74:6379 fd=57 name= age=583 idle=583 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=expire user=default redir=-1
id=3692122 addr=10.0.3.88:53918 laddr=10.0.3.74:6379 fd=112 name= age=127 idle=0 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=8125 qbuf-free=41021 argv-mem=36 obl=0 oll=0 omem=0 tot-mem=69700 events=r cmd=get user=default redir=-1
id=3694805 addr=10.0.3.72:23134 laddr=10.0.3.74:6379 fd=64 name= age=116 idle=2 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61824 events=r cmd=mget user=default redir=-1
id=3694825 addr=10.0.3.88:55192 laddr=10.0.3.74:6379 fd=20 name= age=116 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61824 events=r cmd=mget user=default redir=-1
id=3719469 addr=10.0.3.70:42832 laddr=10.0.3.74:6379 fd=33 name= age=3 idle=2 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719638 addr=10.0.3.43:29714 laddr=10.0.3.74:6379 fd=102 name= age=2 idle=2 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719647 addr=10.0.3.82:59192 laddr=10.0.3.74:6379 fd=55 name= age=2 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719668 addr=10.0.3.82:59208 laddr=10.0.3.74:6379 fd=120 name= age=2 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61824 events=r cmd=mget user=default redir=-1
id=3719701 addr=10.0.3.84:39390 laddr=10.0.3.74:6379 fd=10 name= age=2 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=hgetall user=default redir=-1
id=3719702 addr=10.0.3.29:53230 laddr=10.0.3.74:6379 fd=74 name= age=2 idle=2 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719770 addr=10.0.3.73:32456 laddr=10.0.3.74:6379 fd=111 name= age=2 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719772 addr=10.0.3.71:42536 laddr=10.0.3.74:6379 fd=122 name= age=2 idle=2 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719849 addr=10.0.3.29:53336 laddr=10.0.3.74:6379 fd=22 name= age=2 idle=0 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61480 events=r cmd=setex user=default redir=-1
id=3719861 addr=10.0.3.43:29836 laddr=10.0.3.74:6379 fd=36 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=hgetall user=default redir=-1
id=3719865 addr=10.0.3.73:32500 laddr=10.0.3.74:6379 fd=71 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719870 addr=10.0.3.73:32508 laddr=10.0.3.74:6379 fd=41 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719871 addr=10.0.3.102:32664 laddr=10.0.3.74:6379 fd=8 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719872 addr=10.0.3.73:32510 laddr=10.0.3.74:6379 fd=51 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719875 addr=10.0.3.85:53794 laddr=10.0.3.74:6379 fd=63 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61480 events=r cmd=setex user=default redir=-1
id=3719879 addr=10.0.3.73:32514 laddr=10.0.3.74:6379 fd=9 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=hgetall user=default redir=-1
id=3719881 addr=10.0.3.102:32668 laddr=10.0.3.74:6379 fd=49 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=hgetall user=default redir=-1
id=3719893 addr=10.0.3.71:42604 laddr=10.0.3.74:6379 fd=11 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719894 addr=10.0.3.81:59864 laddr=10.0.3.74:6379 fd=45 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719910 addr=10.0.3.67:31882 laddr=10.0.3.74:6379 fd=87 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719919 addr=10.0.3.67:31896 laddr=10.0.3.74:6379 fd=26 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719922 addr=10.0.3.66:35468 laddr=10.0.3.74:6379 fd=108 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719938 addr=10.0.3.70:43182 laddr=10.0.3.74:6379 fd=16 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=hgetall user=default redir=-1
id=3719941 addr=10.0.3.86:20028 laddr=10.0.3.74:6379 fd=81 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719942 addr=10.0.3.83:51864 laddr=10.0.3.74:6379 fd=18 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719944 addr=10.0.3.86:20030 laddr=10.0.3.74:6379 fd=17 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719959 addr=10.0.3.70:43202 laddr=10.0.3.74:6379 fd=54 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719964 addr=10.0.3.29:53396 laddr=10.0.3.74:6379 fd=116 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=hgetall user=default redir=-1
id=3719967 addr=10.0.3.70:43206 laddr=10.0.3.74:6379 fd=89 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719981 addr=10.0.3.83:51890 laddr=10.0.3.74:6379 fd=40 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=hgetall user=default redir=-1
id=3719985 addr=10.0.3.83:51892 laddr=10.0.3.74:6379 fd=107 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3719986 addr=10.0.3.83:51894 laddr=10.0.3.74:6379 fd=98 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3719991 addr=10.0.3.29:53424 laddr=10.0.3.74:6379 fd=77 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720025 addr=10.0.3.69:46596 laddr=10.0.3.74:6379 fd=117 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720026 addr=10.0.3.82:59360 laddr=10.0.3.74:6379 fd=118 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720037 addr=10.0.3.69:46602 laddr=10.0.3.74:6379 fd=127 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3720038 addr=10.0.3.86:20088 laddr=10.0.3.74:6379 fd=128 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720043 addr=10.0.3.82:59364 laddr=10.0.3.74:6379 fd=46 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3720058 addr=10.0.3.70:43266 laddr=10.0.3.74:6379 fd=60 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720065 addr=10.0.3.83:51962 laddr=10.0.3.74:6379 fd=104 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720068 addr=10.0.3.66:35578 laddr=10.0.3.74:6379 fd=119 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720070 addr=10.0.3.70:43278 laddr=10.0.3.74:6379 fd=97 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3720074 addr=10.0.3.102:32782 laddr=10.0.3.74:6379 fd=103 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720088 addr=10.0.3.43:30002 laddr=10.0.3.74:6379 fd=21 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3720089 addr=10.0.3.102:32788 laddr=10.0.3.74:6379 fd=94 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3720090 addr=10.0.3.71:42708 laddr=10.0.3.74:6379 fd=99 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61824 events=r cmd=mget user=default redir=-1
id=3720092 addr=10.0.3.83:51984 laddr=10.0.3.74:6379 fd=121 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3720093 addr=10.0.3.87:35386 laddr=10.0.3.74:6379 fd=123 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3720096 addr=10.0.3.68:57184 laddr=10.0.3.74:6379 fd=100 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720097 addr=10.0.3.66:35600 laddr=10.0.3.74:6379 fd=24 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720098 addr=10.0.3.71:42718 laddr=10.0.3.74:6379 fd=47 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720099 addr=10.0.3.73:32590 laddr=10.0.3.74:6379 fd=75 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61824 events=r cmd=mget user=default redir=-1
id=3720100 addr=10.0.3.66:35606 laddr=10.0.3.74:6379 fd=83 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720101 addr=10.0.3.69:46644 laddr=10.0.3.74:6379 fd=88 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720103 addr=10.0.3.68:57192 laddr=10.0.3.74:6379 fd=131 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720104 addr=10.0.3.82:59382 laddr=10.0.3.74:6379 fd=133 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720105 addr=10.0.3.69:46648 laddr=10.0.3.74:6379 fd=134 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61824 events=r cmd=mget user=default redir=-1
id=3720106 addr=10.0.3.84:39622 laddr=10.0.3.74:6379 fd=29 name= age=1 idle=0 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=expire user=default redir=-1
id=3720107 addr=10.0.3.84:39624 laddr=10.0.3.74:6379 fd=79 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61824 events=r cmd=mget user=default redir=-1
id=3720108 addr=10.0.3.69:46652 laddr=10.0.3.74:6379 fd=135 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720109 addr=10.0.3.84:39626 laddr=10.0.3.74:6379 fd=136 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720110 addr=10.0.3.87:35398 laddr=10.0.3.74:6379 fd=137 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720111 addr=10.0.3.66:35614 laddr=10.0.3.74:6379 fd=138 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=exists user=default redir=-1
id=3720112 addr=10.0.3.69:46658 laddr=10.0.3.74:6379 fd=105 name= age=1 idle=1 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720113 addr=10.0.3.85:53896 laddr=10.0.3.74:6379 fd=43 name= age=0 idle=0 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61824 events=r cmd=mget user=default redir=-1
id=3720114 addr=10.0.3.43:30016 laddr=10.0.3.74:6379 fd=66 name= age=0 idle=0 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720115 addr=10.0.3.68:57204 laddr=10.0.3.74:6379 fd=73 name= age=0 idle=0 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720116 addr=10.0.3.84:39630 laddr=10.0.3.74:6379 fd=125 name= age=0 idle=0 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=3720117 addr=10.0.3.71:42726 laddr=10.0.3.74:6379 fd=139 name= age=0 idle=0 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61824 events=r cmd=mget user=default redir=-1
id=3720118 addr=10.0.3.87:35402 laddr=10.0.3.74:6379 fd=141 name= age=0 idle=0 flags=N db=11 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61824 events=r cmd=mget user=default redir=-1

------ MODULES INFO OUTPUT ------

uploaded full crash log as file due to report size limitations:

https://github.com/redis/redis/files/8112557/redis-crash-report.log

Additional information

OS distribution and version - CentOS Linux release 7.9.2009 (Core) 3.10.0-1160.53.1.el7.x86_64
no specific steps to reproduce, 16Gb of memory with 4gb limit in redis.conf settings, far away from using all the memory.

Comment From: oranagra

I've edited the post above to include the relevant info from the crash log. this seems a bit similar to #10234

Comment From: yoav-steinberg

@ddrdikko Thanks for the report. Two question: 1. Was this a one off incident or does this happen once in a while? If so how frequently does it happen? 2. Did this crash start happening after some version upgrade, system update or some other change in the environment?

Comment From: ddrdikko

we've upgraded from old stable 3rd version directly to 6.2.6 (completely new clean instance based on latest centos7), it's a clean instance without rdb or something just for caching. problem appeared several times after about 4-6 hours of work (so any usual test could not find any problem) on 2 different instances started in one time with more or less equal load. difference in crash time was around 3 minutes between them. i'll try to mirror requests from stable old redis version to make it even more reproducible.

Comment From: oranagra

@ddrdikko Can you share the systemd unit file used to start Redis? Is it possible that you specify some systemd-level memory limit?

also, maybe the system's overcommit_memory is disabled, and some other process allocated big amounts of memory?

Comment From: ddrdikko

Overcommit_memory was set to 2, but there were no other processes. it's a fresh clean vm dedicated only to this redis instance. According to monitoring, there was 12-13Gb free at the moment of crash.

unit file, no limits specified by me anywhere.

[Unit]
Description=Redis persistent key-value database
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/bin/redis-server /etc/redis/redis.conf --daemonize no --supervised systemd
ExecStop=/usr/libexec/redis-shutdown
Type=notify
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target

Comment From: oranagra

@jasone can you think of any reason why malloc will return NULL in this case, or any other information to we can ask in order to get to the bottom of this.

Some summary: this issue is on a fresh clean centos7 VM with jemalloc 5.1.0. And we have another recent report (#10234), that started after upgrading from Ubuntu 16.04 to 20.04 (happens with both redis 5.0 using jemalloc 6.2.1, and redis 6.2 using jemalloc 5.1.0)

Comment From: ddrdikko

i'll try to look at vm.max_map_count value increasing.

upd: in crash report there was 4084 memory fragments logged. it's close to 4096 possible limit somewhere.

and from ipcs -l output i've got

------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 67108864 max total shared memory (kbytes) = 17179869184 min seg size (bytes) = 1

can it be the cause ?

Comment From: oranagra

@ddrdikko your edit didn't trigger a notification, so we missed it. this seems close enough to be the cause, can you try to increase it and see if the problem is gone?

Comment From: ddrdikko

@oranagra setting kernel.shmmni = 32768 solved my issue. 1 week of tests was done without any issues. maybe this sould be included in administrators/installation guide.