Describe the bug
Currently we are running a calculation which utilizes redis as a cache heavily. We observed a continous linear RSS memory increase whilst the actual datasize stays approx. constant.
I exported the memory info and put them in a table to make analysis a little bit easier. Basically overhead.total increased by 50%
Snapshot 1 Time: 13.03.23 16:35 Snapshot 2 Time: 14.03.23 09:21
MEMORY STATS Snapshot 1
peak.allocated: 2171416048
total.allocated: 2137514456
startup.allocated: 802960
replication.backlog: 0
clients.slaves: 0
clients.normal: 38005994
aof.buffer: 0
lua.caches: 2432
+ db.0
+ -- overhead.hashtable.main: 2984528
+ -- overhead.hashtable.expires: 28904
overhead.total: 41824818
keys.count: 61506
keys.bytes-per-key: 34739
dataset.bytes: 2095689638
dataset.percentage: 98.080131530761719
peak.percentage: 98.438735961914062
allocator.allocated: 2138532616
allocator.active: 2146660352
allocator.resident: 2191577088
allocator-fragmentation.ratio: 1.003800630569458
allocator-fragmentation.bytes: 8127736
allocator-rss.ratio: 1.0209239721298218
allocator-rss.bytes: 44916736
rss-overhead.ratio: 0.94216835498809814
rss-overhead.bytes: -126742528
fragmentation: 0.96561473608016968
fragmentation.bytes: -73528208
*MEMORY STATS Snapshot 2
peak.allocated: 3202992400
total.allocated: 3150967712
startup.allocated: 802960
replication.backlog: 0
clients.slaves: 0
clients.normal: 1051713729
aof.buffer: 0
lua.caches: 2888
+ db.0
+ -- overhead.hashtable.main: 2984408
+ -- overhead.hashtable.expires: 28856
overhead.total: 1055532841
keys.count: 61503
keys.bytes-per-key: 51219
dataset.bytes: 2095434871
dataset.percentage: 66.518264770507812
peak.percentage: 98.375755310058594
allocator.allocated: 3150770472
allocator.active: 3369742336
allocator.resident: 3472744448
allocator-fragmentation.ratio: 1.0694979429244995
allocator-fragmentation.bytes: 218971864
allocator-rss.ratio: 1.0305668115615845
allocator-rss.bytes: 103002112
rss-overhead.ratio: 0.9631497859954834
rss-overhead.bytes: -127971328
fragmentation: 1.0616625547409058
fragmentation.bytes: 194268120
INFO Snapshot 1 ( sorry for the missing sections )
redis_version: 6.2.8
redis_git_sha1: 0
redis_git_dirty: 0
redis_build_id: c75912a30b0c1ae5
redis_mode: standalone
os: Linux 4.19.0-23-amd64 x86_64
arch_bits: 64
monotonic_clock: POSIX clock_gettime
multiplexing_api: epoll
atomicvar_api: c11-builtin
gcc_version: 10.2.1
process_id: 1
process_supervised: no
run_id: dfa62d5eead24f75ec863fd02e61b159ecadbf23
tcp_port: 6379
server_time_usec: 1678723087394268
uptime_in_seconds: 869
uptime_in_days: 0
hz: 10
configured_hz: 10
lru_clock: 1001487
executable: /data/redis-server
config_file: /usr/local/etc/redis/redis.conf
io_threads_active: 0
connected_clients: 305
cluster_connections: 0
maxclients: 10000
client_recent_max_input_buffer: 3145802
client_recent_max_output_buffer: 1763344
blocked_clients: 0
tracking_clients: 0
clients_in_timeout_table: 0
used_memory: 2137516200
used_memory_human: 1.99G
used_memory_rss: 2064834560
used_memory_rss_human: 1.92G
used_memory_peak: 2171416048
used_memory_peak_human: 2.02G
used_memory_peak_perc: 98.44%
used_memory_overhead: 41824818
used_memory_startup: 802960
used_memory_dataset: 2095691382
used_memory_dataset_perc: 98.08%
allocator_allocated: 2138532616
allocator_active: 2146660352
allocator_resident: 2191577088
total_system_memory: 33730154496
total_system_memory_human: 31.41G
used_memory_lua: 34816
used_memory_lua_human: 34.00K
used_memory_scripts: 2432
used_memory_scripts_human: 2.38K
number_of_cached_scripts: 4
maxmemory: 8364490752
maxmemory_human: 7.79G
maxmemory_policy: noeviction
allocator_frag_ratio: 1.0
allocator_frag_bytes: 8127736
allocator_rss_ratio: 1.02
allocator_rss_bytes: 44916736
rss_overhead_ratio: 0.94
rss_overhead_bytes: -126742528
mem_fragmentation_ratio: 0.97
mem_fragmentation_bytes: -73528208
mem_not_counted_for_evict: 0
mem_replication_backlog: 0
mem_clients_slaves: 0
mem_clients_normal: 38005994
mem_aof_buffer: 0
mem_allocator: jemalloc-5.1.0
active_defrag_running: 0
lazyfree_pending_objects: 0
lazyfreed_objects: 0
loading: 0
current_cow_size: 0
current_cow_size_age: 0
current_fork_perc: 0.0
current_save_keys_processed: 0
current_save_keys_total: 0
rdb_changes_since_last_save: 9962
rdb_bgsave_in_progress: 0
rdb_last_save_time: 1678723076
rdb_last_bgsave_status: ok
rdb_last_bgsave_time_sec: 9
rdb_current_bgsave_time_sec: -1
rdb_last_cow_size: 106934272
aof_enabled: 0
aof_rewrite_in_progress: 0
aof_rewrite_scheduled: 0
aof_last_rewrite_time_sec: -1
aof_current_rewrite_time_sec: -1
aof_last_bgrewrite_status: ok
aof_last_write_status: ok
aof_last_cow_size: 0
module_fork_in_progress: 0
module_fork_last_cow_size: 0
total_connections_received: 490
total_commands_processed: 1074548
instantaneous_ops_per_sec: 1240
total_net_input_bytes: 14740010314
total_net_output_bytes: 10982283549
instantaneous_input_kbps: 21156.27
instantaneous_output_kbps: 7230.03
rejected_connections: 0
sync_full: 0
sync_partial_ok: 0
sync_partial_err: 0
expired_keys: 0
expired_stale_perc: 0.0
expired_time_cap_reached_count: 0
expire_cycle_cpu_milliseconds: 139
evicted_keys: 0
keyspace_hits: 371273
keyspace_misses: 38535
pubsub_channels: 1
pubsub_patterns: 0
latest_fork_usec: 35464
total_forks: 12
migrate_cached_sockets: 0
slave_expires_tracked_keys: 0
active_defrag_hits: 0
active_defrag_misses: 0
active_defrag_key_hits: 0
active_defrag_key_misses: 0
tracking_total_keys: 0
tracking_total_items: 0
tracking_total_prefixes: 0
unexpected_error_replies: 0
total_error_replies: 4
dump_payload_sanitizations: 0
total_reads_processed: 1715803
total_writes_processed: 624574
io_threaded_reads_processed: 0
io_threaded_writes_processed: 0
role: master
connected_slaves: 0
master_failover_state: no-failover
master_replid: 3845240ced8d1d4404fe4295c4b157da525d7c1b
master_replid2: 0
master_repl_offset: 0
second_repl_offset: -1
repl_backlog_active: 0
repl_backlog_size: 1048576
repl_backlog_first_byte_offset: 0
repl_backlog_histlen: 0
used_cpu_sys: 48.86537
used_cpu_user: 40.990543
used_cpu_sys_children: 18.152843
used_cpu_user_children: 68.047283
used_cpu_sys_main_thread: 48.509433
used_cpu_user_main_thread: 40.902094
errorstat_LOADING: count=2
errorstat_NOSCRIPT: count=2
cluster_enabled: 0
db0: {'keys': 61506, 'expires': 863, 'avg_ttl': 83072526}
INFO Snapshot 2
redis_version: 6.2.8
redis_git_sha1: 0
redis_git_dirty: 0
redis_build_id: c75912a30b0c1ae5
redis_mode: standalone
os: Linux 4.19.0-23-amd64 x86_64
arch_bits: 64
monotonic_clock: POSIX clock_gettime
multiplexing_api: epoll
atomicvar_api: c11-builtin
gcc_version: 10.2.1
process_id: 1
process_supervised: no
run_id: dfa62d5eead24f75ec863fd02e61b159ecadbf23
tcp_port: 6379
server_time_usec: 1678782098189590
uptime_in_seconds: 59880
uptime_in_days: 0
hz: 10
configured_hz: 10
lru_clock: 1060498
executable: /data/redis-server
config_file: /usr/local/etc/redis/redis.conf
io_threads_active: 0
connected_clients: 321
cluster_connections: 0
maxclients: 10000
client_recent_max_input_buffer: 3670048
client_recent_max_output_buffer: 146706120
blocked_clients: 0
tracking_clients: 0
clients_in_timeout_table: 0
used_memory: 3150969432
used_memory_human: 2.93G
used_memory_rss: 3344773120
used_memory_rss_human: 3.12G
used_memory_peak: 3202992400
used_memory_peak_human: 2.98G
used_memory_peak_perc: 98.38%
used_memory_overhead: 1055532841
used_memory_startup: 802960
used_memory_dataset: 2095436591
used_memory_dataset_perc: 66.52%
allocator_allocated: 3150770472
allocator_active: 3369742336
allocator_resident: 3472744448
total_system_memory: 33730154496
total_system_memory_human: 31.41G
used_memory_lua: 46080
used_memory_lua_human: 45.00K
used_memory_scripts: 2888
used_memory_scripts_human: 2.82K
number_of_cached_scripts: 5
maxmemory: 8364490752
maxmemory_human: 7.79G
maxmemory_policy: noeviction
allocator_frag_ratio: 1.07
allocator_frag_bytes: 218971864
allocator_rss_ratio: 1.03
allocator_rss_bytes: 103002112
rss_overhead_ratio: 0.96
rss_overhead_bytes: -127971328
mem_fragmentation_ratio: 1.06
mem_fragmentation_bytes: 194268120
mem_not_counted_for_evict: 0
mem_replication_backlog: 0
mem_clients_slaves: 0
mem_clients_normal: 1051713729
mem_aof_buffer: 0
mem_allocator: jemalloc-5.1.0
active_defrag_running: 0
lazyfree_pending_objects: 0
lazyfreed_objects: 0
loading: 0
current_cow_size: 1773568
current_cow_size_age: 1
current_fork_perc: 11.66
current_save_keys_processed: 7169
current_save_keys_total: 61498
rdb_changes_since_last_save: 33582
rdb_bgsave_in_progress: 1
rdb_last_save_time: 1678782035
rdb_last_bgsave_status: ok
rdb_last_bgsave_time_sec: 8
rdb_current_bgsave_time_sec: 2
rdb_last_cow_size: 115314688
aof_enabled: 0
aof_rewrite_in_progress: 0
aof_rewrite_scheduled: 0
aof_last_rewrite_time_sec: -1
aof_current_rewrite_time_sec: -1
aof_last_bgrewrite_status: ok
aof_last_write_status: ok
aof_last_cow_size: 0
module_fork_in_progress: 0
module_fork_last_cow_size: 0
total_connections_received: 15708
total_commands_processed: 82306422
instantaneous_ops_per_sec: 1329
total_net_input_bytes: 1227989383766
total_net_output_bytes: 848241611060
instantaneous_input_kbps: 14695.17
instantaneous_output_kbps: 11746.7
rejected_connections: 0
sync_full: 0
sync_partial_ok: 0
sync_partial_err: 0
expired_keys: 36
expired_stale_perc: 0.0
expired_time_cap_reached_count: 0
expire_cycle_cpu_milliseconds: 4968
evicted_keys: 0
keyspace_hits: 28379169
keyspace_misses: 2991991
pubsub_channels: 11
pubsub_patterns: 0
latest_fork_usec: 91059
total_forks: 866
migrate_cached_sockets: 0
slave_expires_tracked_keys: 0
active_defrag_hits: 0
active_defrag_misses: 0
active_defrag_key_hits: 0
active_defrag_key_misses: 0
tracking_total_keys: 0
tracking_total_items: 0
tracking_total_prefixes: 0
unexpected_error_replies: 0
total_error_replies: 4
dump_payload_sanitizations: 0
total_reads_processed: 136569633
total_writes_processed: 45661067
io_threaded_reads_processed: 0
io_threaded_writes_processed: 0
role: master
connected_slaves: 0
master_failover_state: no-failover
master_replid: 3845240ced8d1d4404fe4295c4b157da525d7c1b
master_replid2: 0
master_repl_offset: 0
second_repl_offset: -1
repl_backlog_active: 0
repl_backlog_size: 1048576
repl_backlog_first_byte_offset: 0
repl_backlog_histlen: 0
used_cpu_sys: 3686.542726
used_cpu_user: 2938.010898
used_cpu_sys_children: 1327.033731
used_cpu_user_children: 4997.075887
used_cpu_sys_main_thread: 3656.025538
used_cpu_user_main_thread: 2931.992832
errorstat_LOADING: count=2
errorstat_NOSCRIPT: count=2
cluster_enabled: 0
db0: {'keys': 61503, 'expires': 861, 'avg_ttl': 30566855}
MEMORY MALLOC-STATS Snapshot 1
___ Begin jemalloc statistics ___
Version: "5.1.0-0-g0"
Build-time option settings
config.cache_oblivious: true
config.debug: false
config.fill: true
config.lazy_lock: false
config.malloc_conf: ""
config.prof: false
config.prof_libgcc: false
config.prof_libunwind: false
config.stats: true
config.utrace: false
config.xmalloc: false
Run-time option settings
opt.abort: false
opt.abort_conf: false
opt.retain: true
opt.dss: "secondary"
opt.narenas: 64
opt.percpu_arena: "disabled"
opt.metadata_thp: "disabled"
opt.background_thread: false (background_thread: true)
opt.dirty_decay_ms: 10000 (arenas.dirty_decay_ms: 10000)
opt.muzzy_decay_ms: 10000 (arenas.muzzy_decay_ms: 10000)
opt.junk: "false"
opt.zero: false
opt.tcache: true
opt.lg_tcache_max: 15
opt.thp: "default"
opt.stats_print: false
opt.stats_print_opts: ""
Arenas: 64
Quantum size: 8
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 39
Number of thread-cache bin size classes: 44
Number of large size classes: 196
Allocated: 2137663872, active: 2145767424, metadata: 9032616 (n_thp 0), resident: 2191577088, mapped: 2195152896, retained: 615555072
Background threads: 1, num_runs: 1049, run_interval: 826943681 ns
n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
background_thread 17224 0 0 1 0 0 0
ctl 39089 0 0 1 0 0 0
prof 0 0 0 0 0 0 0
arenas[0]:
assigned threads: 1
uptime: 868823291700
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 10000 5428 723 11541 890756
muzzy: 10000 3557 844 8501 629178
allocated nmalloc ndalloc nrequests
small: 84388224 1585130 97222 10220377
large: 2053275648 172949 171029 172949
total: 2137663872 1758079 268251 10393326
active: 2145767424
mapped: 2195152896
retained: 615555072
base: 8999840
internal: 32776
metadata_thp: 0
tcache_bytes: 14232
resident: 2191577088
n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
large 9859 0 0 1 0 0 0
extent_avail 171058 0 1 1881 0 0 0
extents_dirty 513249 0 18 1463 0 0 0
extents_muzzy 38437 0 1 1857 0 0 0
extents_retained 35133 0 0 1695 0 0 0
decay_dirty 13327 0 0 2109 0 0 0
decay_muzzy 13419 0 0 2095 0 0 0
base 28308 0 0 3 0 0 0
tcache_list 8618 0 0 1 0 0 0
bins: size ind allocated nmalloc ndalloc nrequests curregs curslabs regs pgs util nfills nflushes nslabs nreslabs n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
8 0 1309200 170618 6968 777151 163650 320 512 1 0.998 4618 983 322 171 14543 0 0 1 0 0 0
16 1 6908528 438483 6700 1866585 431783 1687 256 1 0.999 7232 942 1689 2187 18482 0 0 1 0 0 0
24 2 10869168 468100 15218 2938446 452882 885 512 3 0.999 10211 940 886 1322 20655 0 0 1 0 0 0
32 3 1539232 55189 7088 1203928 48101 377 128 1 0.996 2085 975 381 2163 12062 0 0 1 0 0 0
40 4 6379200 161451 1971 874364 159480 312 512 5 0.998 2353 830 312 707 12112 0 0 1 0 0 0
48 5 48000 7321 6321 226698 1000 5 256 3 0.781 2163 942 16 380 11749 0 0 1 0 0 0
56 6 185920 39743 36423 720270 3320 7 512 7 0.926 31583 1071 7 293 41278 0 0 1 0 0 0
64 7 5810048 96149 5367 735530 90782 1420 64 1 0.998 4182 921 1425 2604 15150 0 0 1 0 0 0
80 8 56320 2134 1430 120892 704 3 256 5 0.916 890 746 3 0 10256 0 0 1 0 0 0
96 9 2988096 32151 1025 215281 31126 244 128 3 0.996 762 428 257 43 10077 0 0 1 0 0 0
112 10 7056 370 307 137610 63 1 256 7 0.246 191 191 1 0 9000 0 0 1 0 0 0
128 11 3836288 30097 126 155472 29971 937 32 1 0.999 985 33 939 19 10576 0 0 1 0 0 0
160 12 4796160 31303 1327 101303 29976 235 128 5 0.996 1008 658 235 0 10518 0 0 1 0 0 0
192 13 78720 752 342 17621 410 7 64 3 0.915 113 79 11 8 8824 0 0 1 0 0 0
224 14 6048 1354 1327 22575 27 1 128 7 0.210 739 721 1 0 10078 0 0 1 0 0 0
256 15 256 35 34 24 1 1 16 1 0.062 16 19 14 0 8679 0 0 1 0 0 0
320 16 37440 1560 1443 3316 117 2 64 5 0.914 786 774 3 1 10181 0 0 1 0 0 0
384 17 11136 1838 1809 33146 29 1 32 3 0.906 818 783 67 72 10351 0 0 1 0 0 0
448 18 13410432 29971 37 30111 29934 468 64 7 0.999 478 17 468 0 9580 0 0 1 0 0 0
512 19 512 20 19 12 1 1 8 1 0.125 6 9 2 1 8635 0 0 1 0 0 0
640 20 56320 280 192 8581 88 3 32 5 0.916 71 75 7 2 8774 0 0 1 0 0 0
768 21 2304 641 638 15838 3 1 16 3 0.187 623 624 3 0 9869 0 0 1 0 0 0
896 22 0 100 100 85 0 0 32 7 1 60 64 60 0 8861 0 0 1 0 0 0
1024 23 6144 26 20 14 6 2 4 1 0.750 11 12 6 0 8650 0 0 1 0 0 0
1280 24 2560 133 131 72 2 1 16 5 0.125 61 63 4 0 8748 0 0 1 0 0 0
1536 25 0 17 17 8 0 0 8 3 1 6 8 7 0 8645 0 0 1 0 0 0
1792 26 25856768 14668 239 14834 14429 902 16 7 0.999 1267 125 914 15 10935 0 0 1 0 0 0
2048 27 8192 22 18 12 4 2 2 1 1 8 9 10 5 8652 0 0 1 0 0 0
2560 28 0 70 70 123 0 0 8 5 1 60 61 60 0 8858 0 0 1 0 0 0
3072 29 0 14 14 5 0 0 4 3 1 4 6 6 0 8639 0 0 1 0 0 0
3584 30 0 22 22 13 0 0 8 7 1 8 10 9 0 8653 0 0 1 0 0 0
4096 31 12288 138 135 71 3 3 1 1 1 67 69 138 0 9026 0 0 1 0 0 0
5120 32 0 21 21 7 0 0 4 5 1 7 9 9 0 8651 0 0 1 0 0 0
6144 33 6144 21 20 7 1 1 2 3 0.500 7 8 11 0 8653 0 0 1 0 0 0
7168 34 14336 90 88 184 2 1 4 7 0.500 74 76 28 0 8822 0 0 1 0 0 0
8192 35 16384 62 60 62 2 2 1 2 1 46 48 62 0 8833 0 0 1 0 0 0
10240 36 61440 98 92 85 6 3 2 5 1 76 74 72 6 8908 0 0 1 0 0 0
12288 37 24576 40 38 29 2 2 1 3 1 18 20 40 0 8733 0 0 1 0 0 0
14336 38 43008 28 25 12 3 2 2 7 0.750 11 12 15 5 8668 0 0 1 0 0 0
large: size ind allocated nmalloc ndalloc nrequests curlextents
16384 39 32768 15 13 19 2
20480 40 17879040 3790 2917 8341 873
24576 41 24576 60 59 236 1
28672 42 143360 637 632 1762 5
32768 43 98304 76 73 355 3
40960 44 10608640 70166 69907 70166 259
49152 45 638976 2908 2895 2908 13
57344 46 688128 2530 2518 2530 12
65536 47 1048576 2186 2170 2186 16
81920 48 1392640 3213 3196 3213 17
98304 49 3932160 4836 4796 4836 40
114688 50 802816 1803 1796 1803 7
131072 51 4063232 1604 1573 1604 31
163840 52 5898240 4721 4685 4721 36
196608 53 7274496 7428 7391 7428 37
229376 54 6193152 5755 5728 5755 27
262144 55 1310720 5595 5590 5595 5
327680 56 18350080 6231 6175 6231 56
393216 57 20054016 10330 10279 10330 51
458752 58 13762560 4116 4086 4116 30
524288 59 20447232 4565 4526 4565 39
655360 60 46530560 6988 6917 6988 71
786432 61 19660800 6707 6682 6707 25
917504 62 30277632 2723 2690 2723 33
1048576 63 12582912 2003 1991 2003 12
1310720 64 19660800 3296 3281 3296 15
1572864 65 11010048 2059 2052 2059 7
1835008 66 11010048 2824 2818 2824 6
2097152 67 46137344 1231 1209 1231 22
2621440 68 55050240 1318 1297 1318 21
3145728 69 44040192 951 937 951 14
3670016 70 33030144 88 79 88 9
4194304 71 37748736 19 10 19 9
5242880 72 73400320 24 10 24 14
6291456 73 25165824 14 10 14 4
7340032 74 117440512 24 8 24 16
8388608 75 58720256 8 1 8 7
10485760 76 293601280 39 11 39 28
12582912 77 113246208 14 5 14 9
14680064 78 102760448 11 4 11 7
16777216 79 83886080 9 4 9 5
20971520 80 230686720 17 6 17 11
25165824 81 100663296 6 2 6 4
29360128 82 234881024 8 0 8 8
33554432 83 33554432 1 0 1 1
41943040 84 83886080 2 0 2 2
---
--- End jemalloc statistics ---
MEMORY MALLOC-STATS Snapshot 2
___ Begin jemalloc statistics ___
Version: "5.1.0-0-g0"
Build-time option settings
config.cache_oblivious: true
config.debug: false
config.fill: true
config.lazy_lock: false
config.malloc_conf: ""
config.prof: false
config.prof_libgcc: false
config.prof_libunwind: false
config.stats: true
config.utrace: false
config.xmalloc: false
Run-time option settings
opt.abort: false
opt.abort_conf: false
opt.retain: true
opt.dss: "secondary"
opt.narenas: 64
opt.percpu_arena: "disabled"
opt.metadata_thp: "disabled"
opt.background_thread: false (background_thread: true)
opt.dirty_decay_ms: 10000 (arenas.dirty_decay_ms: 10000)
opt.muzzy_decay_ms: 10000 (arenas.muzzy_decay_ms: 10000)
opt.junk: "false"
opt.zero: false
opt.tcache: true
opt.lg_tcache_max: 15
opt.thp: "default"
opt.stats_print: false
opt.stats_print_opts: ""
Arenas: 64
Quantum size: 8
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 39
Number of thread-cache bin size classes: 44
Number of large size classes: 196
Allocated: 3151213128, active: 3370209280, metadata: 67730296 (n_thp 0), resident: 3472744448, mapped: 3480543232, retained: 24357576704
Background threads: 1, num_runs: 82086, run_interval: 728931891 ns
n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
background_thread 1185291 0 0 1 0 0 0
ctl 2379029 0 0 1 0 0 0
prof 0 0 0 0 0 0 0
arenas[0]:
assigned threads: 1
uptime: 59879613065152
dss allocation precedence: "secondary"
decaying: time npages sweeps madvises purged
dirty: 10000 4495 56873 896555 74244656
muzzy: 10000 4010 67341 663612 53316768
allocated nmalloc ndalloc nrequests
small: 79131208 8623188 7108431 613373683
large: 3072081920 13074594 13022779 13074594
total: 3151213128 21697782 20131210 626448277
active: 3370209280
mapped: 3480543232
retained: 24357576704
base: 67697520
internal: 32776
metadata_thp: 0
tcache_bytes: 93872
resident: 3472744448
n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
large 692483 0 0 1 0 0 0
extent_avail 12722790 5 140 152199 0 0 1
extents_dirty 38323875 73 1265 114953 148001229 28000231 1
extents_muzzy 2615865 1 14 146997 4000033 4000033 1
extents_retained 2095335 5 21 135081 20000169 20000169 1
decay_dirty 944999 0 2 165147 0 0 0
decay_muzzy 955404 0 6 164641 0 0 0
base 1294523 0 0 3 0 0 0
tcache_list 593078 0 0 1 0 0 0
bins: size ind allocated nmalloc ndalloc nrequests curregs curslabs regs pgs util nfills nflushes nslabs nreslabs n_lock_ops n_waiting n_spin_acq n_owner_switch total_wait_ns max_wait_ns max_n_thds
8 0 1228888 654058 500447 42004544 153611 320 512 1 0.937 225661 75625 322 265528 894688 0 0 1 0 0 0
16 1 6909152 922126 490304 92597248 431822 1688 256 1 0.999 198986 71176 1696 132692 864943 0 0 1 0 0 0
24 2 11910888 1644567 1148280 185240125 496287 970 512 3 0.999 437897 71565 1127 236604 1103823 0 0 1 0 0 0
32 3 1861152 557074 498913 82321583 58161 455 128 1 0.998 122973 73199 800 111358 790394 0 0 1 0 0 0
40 4 6007880 291051 140854 48580585 150197 302 512 5 0.971 57136 64255 312 58038 714790 0 0 1 0 0 0
48 5 51072 445558 444494 14322945 1064 5 256 3 0.831 168867 70736 16 29639 832707 0 0 1 0 0 0
56 6 153888 2811468 2808720 55521778 2748 7 512 7 0.766 2566284 74577 7 45900 3233945 0 0 1 0 0 0
64 7 5596416 480263 392819 45596519 87444 1418 64 1 0.963 203967 69177 1425 352136 867653 0 0 1 0 0 0
80 8 45200 94279 93714 7043811 565 3 256 5 0.735 58744 53798 3 0 705622 0 0 1 0 0 0
96 9 2989248 76443 45305 12145149 31138 244 128 3 0.996 32829 31944 257 43 658120 0 0 1 0 0 0
112 10 5488 15144 15095 6127351 49 1 256 7 0.191 14863 14842 1 0 622783 0 0 1 0 0 0
128 11 3835904 33292 3324 8862345 29968 937 32 1 0.999 3733 2799 939 2664 600550 0 0 1 0 0 0
160 12 4796000 115085 85110 3294246 29975 235 128 5 0.996 53939 50633 235 2794 697884 0 0 1 0 0 0
192 13 78720 770 360 4169510 410 7 64 3 0.915 130 97 11 8 593319 0 0 1 0 0 0
224 14 6272 85319 85291 1758284 28 1 128 7 0.218 51865 50562 1 0 695505 0 0 1 0 0 0
256 15 256 53 52 39 1 1 16 1 0.062 31 37 14 0 593172 0 0 1 0 0 0
320 16 37120 93188 93072 249118 116 2 64 5 0.906 54852 52489 28 18229 700472 0 0 1 0 0 0
384 17 11136 134118 134089 277916 29 1 32 3 0.906 61656 58910 5261 5909 724164 0 0 1 0 0 0
448 18 13410432 33142 3208 33463 29934 468 64 7 0.999 3649 3188 468 0 600382 0 0 1 0 0 0
512 19 512 3224 3223 3364 1 1 8 1 0.125 3210 3213 2 1 599503 0 0 1 0 0 0
640 20 56320 306 218 2980247 88 3 32 5 0.916 85 90 7 2 593263 0 0 1 0 0 0
768 21 2304 49072 49069 131884 3 1 16 3 0.187 49020 49024 3 0 691126 0 0 1 0 0 0
896 22 0 101 101 86 0 0 32 7 1 61 65 61 0 593325 0 0 1 0 0 0
1024 23 6144 26 20 14 6 2 4 1 0.750 11 12 6 0 593110 0 0 1 0 0 0
1280 24 2560 8003 8001 4007 2 1 16 5 0.125 3996 3998 4 0 601078 0 0 1 0 0 0
1536 25 0 17 17 8 0 0 8 3 1 6 8 7 0 593105 0 0 1 0 0 0
1792 26 19851776 19068 7990 25471 11078 902 16 7 0.767 5667 7837 914 1080 607507 0 0 1 0 0 0
2048 27 8192 24 20 13 4 2 2 1 1 9 10 11 5 593116 0 0 1 0 0 0
2560 28 0 4005 4005 7993 0 0 8 5 1 3995 3996 3995 0 609058 0 0 1 0 0 0
3072 29 0 14 14 5 0 0 4 3 1 4 6 6 0 593099 0 0 1 0 0 0
3584 30 0 49 49 35 0 0 8 7 1 30 37 23 0 593190 0 0 1 0 0 0
4096 31 12288 4758 4755 4014 3 3 1 1 1 4009 4013 4758 0 610612 0 0 1 0 0 0
5120 32 0 40 40 23 0 0 4 5 1 23 28 21 0 593170 0 0 1 0 0 0
6144 33 18432 39 36 24 3 2 2 3 0.750 23 24 13 1 593148 0 0 1 0 0 0
7168 34 14336 108 106 199 2 1 4 7 0.500 89 94 31 3 593321 0 0 1 0 0 0
8192 35 24576 92 89 84 3 3 1 2 1 69 77 92 0 593404 0 0 1 0 0 0
10240 36 102400 34229 34219 46838 10 6 2 5 0.833 22432 16349 15450 5476 662752 0 0 1 0 0 0
12288 37 24576 1284 1282 4582 2 2 1 3 1 1084 1074 1284 0 597801 0 0 1 0 0 0
14336 38 71680 11731 11726 18233 5 3 2 7 0.833 7702 5634 4942 2981 616294 0 0 1 0 0 0
large: size ind allocated nmalloc ndalloc nrequests curlextents
16384 39 131072 8222 8214 16638 8
20480 40 1039134720 273327 222588 630667 50739
24576 41 0 21940 21940 70361 0
28672 42 143360 19737 19732 66372 5
32768 43 65536 17485 17483 60426 2
40960 44 11550720 5279684 5279402 5279684 282
49152 45 786432 139964 139948 139964 16
57344 46 974848 155261 155244 155261 17
65536 47 458752 129512 129505 129512 7
81920 48 1638400 280281 280261 280281 20
98304 49 3047424 284842 284811 284842 31
114688 50 1605632 239028 239014 239028 14
131072 51 3145728 229799 229775 229799 24
163840 52 6553600 434418 434378 434418 40
196608 53 9240576 409630 409583 409630 47
229376 54 6881280 349705 349675 349705 30
262144 55 1310720 354617 354612 354617 5
327680 56 16711680 632964 632913 632964 51
393216 57 9830400 545840 545815 545840 25
458752 58 17432576 457419 457381 457419 38
524288 59 28311552 376090 376036 376090 54
655360 60 47185920 600689 600617 600689 72
786432 61 26738688 436605 436571 436605 34
917504 62 23855104 299145 299119 299145 26
1048576 63 10485760 213934 213924 213934 10
1310720 64 14417920 308881 308870 308881 11
1572864 65 28311552 191380 191362 191380 18
1835008 66 11010048 119678 119672 119678 6
2097152 67 33554432 72913 72897 72913 16
2621440 68 47185920 115772 115754 115772 18
3145728 69 44040192 41530 41516 41530 14
3670016 70 36700160 15242 15232 15242 10
4194304 71 37748736 6019 6010 6019 9
5242880 72 73400320 5972 5958 5972 14
6291456 73 25165824 4064 4060 4064 4
7340032 74 117440512 1337 1321 1337 16
8388608 75 58720256 522 515 522 7
10485760 76 293601280 822 794 822 28
12582912 77 113246208 270 261 270 9
14680064 78 102760448 11 4 11 7
16777216 79 83886080 9 4 9 5
20971520 80 230686720 17 6 17 11
25165824 81 100663296 6 2 6 4
29360128 82 234881024 8 0 8 8
33554432 83 33554432 1 0 1 1
41943040 84 83886080 2 0 2 2
---
--- End jemalloc statistics ---
Redis Metrics Please not the marked spaces with "False Monitoring". At that time the Redis Container was restarted as it was running close to the configured limit. At that time the monitoring values are wrong as they are "double counted".
To reproduce
I can't really tell. I attached as much information as i could gather. Probably one issue to reproduce is having a heavy R/W load into the redis-database. The data types used are:
hash: 59883 entries, total: 76.26 MB, avg: 1.30 kB/key
string: 1236 entries, total: 1.46 GB, avg: 1.21 MB/key
zset: 278 entries, total: 1.40 MB, avg: 5.15 kB/key
list: 60 entries, total: 355.26 MB, avg: 5.92 MB/key
set: 1 entries, total: 1.17 kB, avg: 1.17 kB/key
Expected behavior
Don't have an increase of memory which is just overhead, expecially not in the range of 50 - 100% of the original dataset
Additional Information
The calculation will be going on for another 60h. I can create another Snapshot of the Stats at that point.
If there is information missing please let me know. Thank you very much
Comment From: sundb
@mspi92 Please don't post the logs as tables that are hard to look at.
It seems that the growth in memory is mainly from the normal clients(clients.normal 38005994 1051713729 1013707735).
Can you also post the output of CLIENT LIST
Comment From: mspi92
@sundb: sorry for that, i corrected the original post and replaced the tables by the "raw" export.
Attached here is the list of clients:
id=19926 addr=10.244.1.33:48810 laddr=10.244.3.221:6379 fd=347 name= age=5250 idle=3203 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hgetall user=default redir=-1
id=91 addr=10.244.3.222:51066 laddr=10.244.3.221:6379 fd=74 name= age=81309 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=262134 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=282640 events=r cmd=get user=default redir=-1
id=42 addr=10.244.3.222:50974 laddr=10.244.3.221:6379 fd=26 name= age=81312 idle=65 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=37 addr=10.244.3.223:54650 laddr=10.244.3.221:6379 fd=21 name= age=81314 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=publish user=default redir=-1
id=50 addr=10.244.1.38:57594 laddr=10.244.3.221:6379 fd=34 name= age=81311 idle=76 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=276 addr=10.244.3.222:52882 laddr=10.244.3.221:6379 fd=255 name= age=81297 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=277 addr=10.244.3.222:52896 laddr=10.244.3.221:6379 fd=256 name= age=81297 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=278 addr=10.244.3.223:46260 laddr=10.244.3.221:6379 fd=257 name= age=81297 idle=26 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1
id=139 addr=10.244.4.16:60182 laddr=10.244.3.221:6379 fd=121 name= age=81307 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=19712 addr=10.244.4.16:34254 laddr=10.244.3.221:6379 fd=330 name= age=5896 idle=5071 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=140 addr=10.244.1.38:58394 laddr=10.244.3.221:6379 fd=122 name= age=81307 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=155 addr=10.244.4.16:60250 laddr=10.244.3.221:6379 fd=137 name= age=81306 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=327 addr=10.244.4.17:39104 laddr=10.244.3.221:6379 fd=288 name= age=81268 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=169 addr=10.244.4.16:60328 laddr=10.244.3.221:6379 fd=151 name= age=81303 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=19715 addr=10.244.1.33:35556 laddr=10.244.3.221:6379 fd=331 name= age=5895 idle=5235 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=170 addr=10.244.3.222:34184 laddr=10.244.3.221:6379 fd=152 name= age=81303 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16381 oll=1597 omem=32744888 tot-mem=32765392 events=rw cmd=unsubscribe user=default redir=-1
id=171 addr=10.244.4.16:60330 laddr=10.244.3.221:6379 fd=153 name= age=81303 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=387 addr=10.244.2.144:52892 laddr=10.244.3.221:6379 fd=306 name= age=81007 idle=1745 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=251 addr=10.244.4.17:38292 laddr=10.244.3.221:6379 fd=230 name= age=81298 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=252 addr=10.244.1.38:60300 laddr=10.244.3.221:6379 fd=231 name= age=81298 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=1158 addr=10.244.1.38:34208 laddr=10.244.3.221:6379 fd=314 name= age=78008 idle=26348 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=156 addr=10.244.2.145:54570 laddr=10.244.3.221:6379 fd=138 name= age=81306 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=291 addr=10.244.4.17:38394 laddr=10.244.3.221:6379 fd=270 name= age=81296 idle=277 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=26 addr=10.244.1.38:57396 laddr=10.244.3.221:6379 fd=16 name= age=81337 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=85 addr=10.244.2.144:45812 laddr=10.244.3.221:6379 fd=68 name= age=81310 idle=48 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16381 oll=434 omem=8898736 tot-mem=8919240 events=rw cmd=unsubscribe user=default redir=-1
id=86 addr=10.244.3.222:51052 laddr=10.244.3.221:6379 fd=69 name= age=81309 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=19985 addr=10.244.1.33:43570 laddr=10.244.3.221:6379 fd=344 name= age=5059 idle=4977 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=74 addr=10.244.4.16:60000 laddr=10.244.3.221:6379 fd=57 name= age=81310 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=80 addr=10.244.2.145:54332 laddr=10.244.3.221:6379 fd=63 name= age=81310 idle=1 flags=P db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16333 oll=8029 omem=164626616 tot-mem=164647120 events=rw cmd=subscribe user=default redir=-1
id=81 addr=10.244.2.145:54336 laddr=10.244.3.221:6379 fd=64 name= age=81310 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=82 addr=10.244.4.17:37708 laddr=10.244.3.221:6379 fd=65 name= age=81310 idle=17 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=83 addr=10.244.3.222:51038 laddr=10.244.3.221:6379 fd=66 name= age=81310 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=297 addr=10.244.2.145:55030 laddr=10.244.3.221:6379 fd=276 name= age=81295 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=6761 addr=10.244.1.38:44968 laddr=10.244.3.221:6379 fd=320 name= age=56196 idle=56195 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=136 addr=10.244.3.222:34118 laddr=10.244.3.221:6379 fd=118 name= age=81307 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=137 addr=10.244.1.38:58306 laddr=10.244.3.221:6379 fd=119 name= age=81307 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=138 addr=10.244.3.222:34132 laddr=10.244.3.221:6379 fd=120 name= age=81307 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=315 addr=10.244.4.16:60840 laddr=10.244.3.221:6379 fd=296 name= age=81291 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=20320 addr=10.244.1.33:40790 laddr=10.244.3.221:6379 fd=332 name= age=3842 idle=3171 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=zrem user=default redir=-1
id=316 addr=10.244.2.145:55136 laddr=10.244.3.221:6379 fd=297 name= age=81291 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=154 addr=10.244.4.16:60234 laddr=10.244.3.221:6379 fd=136 name= age=81306 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=84 addr=10.244.3.222:51050 laddr=10.244.3.221:6379 fd=67 name= age=81310 idle=28427 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=75 addr=10.244.4.17:37702 laddr=10.244.3.221:6379 fd=58 name= age=81310 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=275 addr=10.244.4.17:38352 laddr=10.244.3.221:6379 fd=254 name= age=81297 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=76 addr=10.244.2.144:45786 laddr=10.244.3.221:6379 fd=59 name= age=81310 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=77 addr=10.244.2.144:45788 laddr=10.244.3.221:6379 fd=60 name= age=81310 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16384 oll=8379 omem=171803016 tot-mem=171823520 events=rw cmd=unsubscribe user=default redir=-1
id=301 addr=10.244.2.145:55062 laddr=10.244.3.221:6379 fd=280 name= age=81294 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=326 addr=10.244.2.145:55704 laddr=10.244.3.221:6379 fd=287 name= age=81268 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=19755 addr=10.244.4.17:43500 laddr=10.244.3.221:6379 fd=335 name= age=5755 idle=5045 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=302 addr=10.244.4.17:38460 laddr=10.244.3.221:6379 fd=281 name= age=81294 idle=8257 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=78 addr=10.244.1.38:57644 laddr=10.244.3.221:6379 fd=61 name= age=81310 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=79 addr=10.244.3.222:51022 laddr=10.244.3.221:6379 fd=62 name= age=81310 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=262134 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=282648 events=r cmd=rpush user=default redir=-1
id=69 addr=10.244.2.144:45784 laddr=10.244.3.221:6379 fd=52 name= age=81310 idle=698 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=249 addr=10.244.1.38:60286 laddr=10.244.3.221:6379 fd=228 name= age=81299 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=5291 addr=10.244.2.145:37950 laddr=10.244.3.221:6379 fd=319 name= age=61905 idle=3477 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=1998 addr=10.244.3.223:37662 laddr=10.244.3.221:6379 fd=315 name= age=74696 idle=63 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=250 addr=10.244.4.16:60590 laddr=10.244.3.221:6379 fd=229 name= age=81299 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=70 addr=10.244.3.222:51010 laddr=10.244.3.221:6379 fd=53 name= age=81310 idle=0 flags=P db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16357 oll=9661 omem=198089144 tot-mem=198109648 events=rw cmd=unsubscribe user=default redir=-1
id=133 addr=10.244.4.17:37866 laddr=10.244.3.221:6379 fd=115 name= age=81307 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=134 addr=10.244.3.222:34104 laddr=10.244.3.221:6379 fd=116 name= age=81307 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=307 addr=10.244.4.16:60790 laddr=10.244.3.221:6379 fd=286 name= age=81293 idle=97 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16336 oll=111 omem=2275944 tot-mem=2296448 events=rw cmd=unsubscribe user=default redir=-1
id=135 addr=10.244.4.17:37870 laddr=10.244.3.221:6379 fd=117 name= age=81307 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=319 addr=10.244.4.16:60888 laddr=10.244.3.221:6379 fd=299 name= age=81289 idle=8257 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=71 addr=10.244.4.17:37690 laddr=10.244.3.221:6379 fd=54 name= age=81310 idle=17 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16336 oll=1698 omem=34815792 tot-mem=34836296 events=rw cmd=unsubscribe user=default redir=-1
id=232 addr=10.244.2.145:54816 laddr=10.244.3.221:6379 fd=213 name= age=81300 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=233 addr=10.244.2.145:54818 laddr=10.244.3.221:6379 fd=214 name= age=81300 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=235 addr=10.244.4.16:60538 laddr=10.244.3.221:6379 fd=216 name= age=81300 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=72 addr=10.244.4.16:59992 laddr=10.244.3.221:6379 fd=55 name= age=81310 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=73 addr=10.244.1.38:57626 laddr=10.244.3.221:6379 fd=56 name= age=81310 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=19922 addr=10.244.1.33:48222 laddr=10.244.3.221:6379 fd=342 name= age=5252 idle=4979 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=34 addr=10.244.3.223:54618 laddr=10.244.3.221:6379 fd=18 name= age=81316 idle=55326 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=296 addr=10.244.2.145:55024 laddr=10.244.3.221:6379 fd=275 name= age=81295 idle=24217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=65 addr=10.244.4.16:59986 laddr=10.244.3.221:6379 fd=48 name= age=81310 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=262134 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=282648 events=r cmd=rpush user=default redir=-1
id=66 addr=10.244.3.222:51000 laddr=10.244.3.221:6379 fd=49 name= age=81310 idle=28427 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=unsubscribe user=default redir=-1
id=19806 addr=10.244.1.38:42890 laddr=10.244.3.221:6379 fd=340 name= age=5621 idle=4978 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=129 addr=10.244.4.16:60160 laddr=10.244.3.221:6379 fd=111 name= age=81307 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=401 addr=10.244.3.222:60620 laddr=10.244.3.221:6379 fd=308 name= age=80948 idle=349 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=272 addr=10.244.1.38:60572 laddr=10.244.3.221:6379 fd=251 name= age=81298 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=273 addr=10.244.4.17:38348 laddr=10.244.3.221:6379 fd=252 name= age=81298 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=20288 addr=10.244.1.33:42688 laddr=10.244.3.221:6379 fd=346 name= age=3952 idle=3950 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=zrem user=default redir=-1
id=274 addr=10.244.2.144:46386 laddr=10.244.3.221:6379 fd=253 name= age=81297 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=130 addr=10.244.2.144:45966 laddr=10.244.3.221:6379 fd=112 name= age=81307 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=149 addr=10.244.4.17:37922 laddr=10.244.3.221:6379 fd=131 name= age=81306 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=299 addr=10.244.2.144:46524 laddr=10.244.3.221:6379 fd=278 name= age=81294 idle=81294 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=19876 addr=10.244.1.33:35302 laddr=10.244.3.221:6379 fd=323 name= age=5390 idle=5311 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=zrem user=default redir=-1
id=19877 addr=10.244.1.33:35306 laddr=10.244.3.221:6379 fd=341 name= age=5390 idle=5311 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=zrem user=default redir=-1
id=19879 addr=10.244.1.33:35318 laddr=10.244.3.221:6379 fd=345 name= age=5390 idle=5081 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=300 addr=10.244.2.145:55060 laddr=10.244.3.221:6379 fd=279 name= age=81294 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=243 addr=10.244.1.38:60268 laddr=10.244.3.221:6379 fd=222 name= age=81299 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=244 addr=10.244.1.38:60272 laddr=10.244.3.221:6379 fd=223 name= age=81299 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=245 addr=10.244.1.38:60274 laddr=10.244.3.221:6379 fd=224 name= age=81299 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=246 addr=10.244.4.16:60584 laddr=10.244.3.221:6379 fd=225 name= age=81299 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=247 addr=10.244.2.144:46324 laddr=10.244.3.221:6379 fd=226 name= age=81299 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=248 addr=10.244.2.144:46326 laddr=10.244.3.221:6379 fd=227 name= age=81299 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=150 addr=10.244.2.144:46024 laddr=10.244.3.221:6379 fd=132 name= age=81306 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=151 addr=10.244.3.222:34174 laddr=10.244.3.221:6379 fd=133 name= age=81306 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=152 addr=10.244.1.38:58660 laddr=10.244.3.221:6379 fd=134 name= age=81306 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=153 addr=10.244.2.145:54562 laddr=10.244.3.221:6379 fd=135 name= age=81306 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=290 addr=10.244.4.17:38388 laddr=10.244.3.221:6379 fd=269 name= age=81297 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=655350 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=675872 events=r cmd=lrange user=default redir=-1
id=131 addr=10.244.2.144:45968 laddr=10.244.3.221:6379 fd=113 name= age=81307 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=132 addr=10.244.4.16:60164 laddr=10.244.3.221:6379 fd=114 name= age=81307 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=67 addr=10.244.1.38:57616 laddr=10.244.3.221:6379 fd=50 name= age=81310 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=68 addr=10.244.1.38:57618 laddr=10.244.3.221:6379 fd=51 name= age=81310 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=12 addr=10.244.3.2:33318 laddr=10.244.3.221:6379 fd=11 name= age=81355 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=slowlog user=default redir=-1
id=35 addr=10.244.3.223:54634 laddr=10.244.3.221:6379 fd=19 name= age=81316 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61496 events=r cmd=eval user=default redir=-1
id=49 addr=10.244.4.16:59962 laddr=10.244.3.221:6379 fd=33 name= age=81311 idle=0 flags=P db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=16384 oll=9537 omem=195546648 tot-mem=195608104 events=rw cmd=subscribe user=default redir=-1
id=2803 addr=10.244.1.38:42888 laddr=10.244.3.221:6379 fd=317 name= age=71538 idle=3476 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=231 addr=10.244.2.145:54810 laddr=10.244.3.221:6379 fd=212 name= age=81300 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=98294 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=118808 events=r cmd=rpush user=default redir=-1
id=314 addr=10.244.3.222:52910 laddr=10.244.3.221:6379 fd=295 name= age=81291 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=1310710 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=1331232 events=r cmd=lrange user=default redir=-1
id=63 addr=10.244.2.145:54314 laddr=10.244.3.221:6379 fd=46 name= age=81310 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16381 oll=562 omem=11523248 tot-mem=11543752 events=rw cmd=unsubscribe user=default redir=-1
id=64 addr=10.244.1.38:57612 laddr=10.244.3.221:6379 fd=47 name= age=81310 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16383 oll=161 omem=3301144 tot-mem=3321648 events=rw cmd=unsubscribe user=default redir=-1
id=400 addr=10.244.2.144:54144 laddr=10.244.3.221:6379 fd=307 name= age=80948 idle=76 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=126 addr=10.244.4.16:60154 laddr=10.244.3.221:6379 fd=108 name= age=81307 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1
id=271 addr=10.244.3.222:52878 laddr=10.244.3.221:6379 fd=250 name= age=81298 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=127 addr=10.244.3.222:34094 laddr=10.244.3.221:6379 fd=109 name= age=81307 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=166 addr=10.244.2.145:54598 laddr=10.244.3.221:6379 fd=148 name= age=81305 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=167 addr=10.244.2.145:54602 laddr=10.244.3.221:6379 fd=149 name= age=81305 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=128 addr=10.244.1.38:58222 laddr=10.244.3.221:6379 fd=110 name= age=81307 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=241 addr=10.244.4.16:60576 laddr=10.244.3.221:6379 fd=220 name= age=81299 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=242 addr=10.244.4.17:38278 laddr=10.244.3.221:6379 fd=221 name= age=81299 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=6 addr=10.244.3.182:58190 laddr=10.244.3.221:6379 fd=8 name= age=81371 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=55 addr=10.244.2.144:45776 laddr=10.244.3.221:6379 fd=38 name= age=81311 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=60 addr=10.244.1.38:57606 laddr=10.244.3.221:6379 fd=43 name= age=81310 idle=2718 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=61 addr=10.244.1.38:57608 laddr=10.244.3.221:6379 fd=44 name= age=81310 idle=1 flags=P db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16383 oll=9337 omem=191445848 tot-mem=191466352 events=rw cmd=unsubscribe user=default redir=-1
id=62 addr=10.244.3.222:50988 laddr=10.244.3.221:6379 fd=45 name= age=81310 idle=44106 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=22 addr=10.244.3.222:55846 laddr=10.244.3.221:6379 fd=12 name= age=81337 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=288 addr=10.244.1.38:60778 laddr=10.244.3.221:6379 fd=267 name= age=81297 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=524278 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=544800 events=r cmd=lrange user=default redir=-1
id=289 addr=10.244.2.145:54954 laddr=10.244.3.221:6379 fd=268 name= age=81297 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=1310710 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=1331232 events=r cmd=lrange user=default redir=-1
id=23 addr=10.244.4.16:59712 laddr=10.244.3.221:6379 fd=13 name= age=81337 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1
id=47 addr=10.244.2.144:45750 laddr=10.244.3.221:6379 fd=31 name= age=81312 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16333 oll=2515 omem=51567560 tot-mem=51588064 events=rw cmd=unsubscribe user=default redir=-1
id=4052 addr=10.244.4.16:56414 laddr=10.244.3.221:6379 fd=318 name= age=66698 idle=13978 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=217 addr=10.244.2.144:46168 laddr=10.244.3.221:6379 fd=199 name= age=81302 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=218 addr=10.244.2.144:46170 laddr=10.244.3.221:6379 fd=200 name= age=81302 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=219 addr=10.244.2.144:46172 laddr=10.244.3.221:6379 fd=201 name= age=81302 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=220 addr=10.244.3.222:34300 laddr=10.244.3.221:6379 fd=202 name= age=81302 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=230 addr=10.244.2.145:54800 laddr=10.244.3.221:6379 fd=211 name= age=81300 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=229366 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=249880 events=r cmd=rpush user=default redir=-1
id=121 addr=10.244.2.145:54446 laddr=10.244.3.221:6379 fd=103 name= age=81309 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=125 addr=10.244.1.38:58202 laddr=10.244.3.221:6379 fd=107 name= age=81308 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=58 addr=10.244.2.145:54312 laddr=10.244.3.221:6379 fd=41 name= age=81310 idle=1055 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=269 addr=10.244.3.222:52876 laddr=10.244.3.221:6379 fd=248 name= age=81298 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=270 addr=10.244.1.38:60496 laddr=10.244.3.221:6379 fd=249 name= age=81298 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=59 addr=10.244.3.222:50986 laddr=10.244.3.221:6379 fd=42 name= age=81310 idle=288 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=19827 addr=10.244.1.33:52638 laddr=10.244.3.221:6379 fd=336 name= age=5556 idle=590 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=exists user=default redir=-1
id=48 addr=10.244.4.17:37660 laddr=10.244.3.221:6379 fd=32 name= age=81311 idle=94 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=14430 addr=10.244.3.222:36294 laddr=10.244.3.221:6379 fd=322 name= age=26476 idle=5222 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=28 addr=10.244.2.145:54106 laddr=10.244.3.221:6379 fd=17 name= age=81332 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1
id=298 addr=10.244.4.17:38448 laddr=10.244.3.221:6379 fd=277 name= age=81295 idle=8257 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=unsubscribe user=default redir=-1
id=239 addr=10.244.4.17:38274 laddr=10.244.3.221:6379 fd=219 name= age=81299 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=458 addr=10.244.2.144:60188 laddr=10.244.3.221:6379 fd=310 name= age=80678 idle=13238 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=223 addr=10.244.2.144:46236 laddr=10.244.3.221:6379 fd=205 name= age=81301 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1
id=9 addr=10.244.3.183:53530 laddr=10.244.3.221:6379 fd=10 name= age=81369 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=57 addr=10.244.4.16:59982 laddr=10.244.3.221:6379 fd=40 name= age=81310 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16383 oll=1705 omem=34959320 tot-mem=34979824 events=rw cmd=unsubscribe user=default redir=-1
id=13040 addr=10.244.4.17:45726 laddr=10.244.3.221:6379 fd=321 name= age=31839 idle=3577 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=123 addr=10.244.4.17:37848 laddr=10.244.3.221:6379 fd=105 name= age=81308 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=124 addr=10.244.2.145:54494 laddr=10.244.3.221:6379 fd=106 name= age=81308 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=119 addr=10.244.2.145:54432 laddr=10.244.3.221:6379 fd=101 name= age=81309 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=655350 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=675864 events=r cmd=rpush user=default redir=-1
id=306 addr=10.244.4.16:60782 laddr=10.244.3.221:6379 fd=285 name= age=81294 idle=278 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=120 addr=10.244.2.145:54434 laddr=10.244.3.221:6379 fd=102 name= age=81309 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=24 addr=10.244.4.17:37418 laddr=10.244.3.221:6379 fd=14 name= age=81337 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=147 addr=10.244.1.38:58546 laddr=10.244.3.221:6379 fd=129 name= age=81307 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=148 addr=10.244.3.222:34158 laddr=10.244.3.221:6379 fd=130 name= age=81306 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=46 addr=10.244.3.222:50976 laddr=10.244.3.221:6379 fd=30 name= age=81312 idle=64 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=779 addr=10.244.2.145:38840 laddr=10.244.3.221:6379 fd=312 name= age=79478 idle=688 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=228 addr=10.244.2.145:54794 laddr=10.244.3.221:6379 fd=209 name= age=81300 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=229 addr=10.244.2.145:54796 laddr=10.244.3.221:6379 fd=210 name= age=81300 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=131062 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=151576 events=r cmd=rpush user=default redir=-1
id=263 addr=10.244.2.145:54900 laddr=10.244.3.221:6379 fd=242 name= age=81298 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=264 addr=10.244.4.16:60638 laddr=10.244.3.221:6379 fd=243 name= age=81298 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=265 addr=10.244.3.222:52860 laddr=10.244.3.221:6379 fd=244 name= age=81298 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=266 addr=10.244.1.38:60458 laddr=10.244.3.221:6379 fd=245 name= age=81298 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=312 addr=10.244.4.16:60826 laddr=10.244.3.221:6379 fd=293 name= age=81292 idle=8257 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=unsubscribe user=default redir=-1
id=313 addr=10.244.2.145:55126 laddr=10.244.3.221:6379 fd=294 name= age=81291 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=267 addr=10.244.2.144:46370 laddr=10.244.3.221:6379 fd=246 name= age=81298 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=268 addr=10.244.3.222:52862 laddr=10.244.3.221:6379 fd=247 name= age=81298 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=38 addr=10.244.4.16:59930 laddr=10.244.3.221:6379 fd=22 name= age=81313 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=165 addr=10.244.2.144:46060 laddr=10.244.3.221:6379 fd=147 name= age=81305 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=786422 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=806944 events=r cmd=lrange user=default redir=-1
id=39 addr=10.244.2.144:45734 laddr=10.244.3.221:6379 fd=23 name= age=81312 idle=74 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=43 addr=10.244.2.145:54278 laddr=10.244.3.221:6379 fd=27 name= age=81312 idle=92 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=52 addr=10.244.2.145:54300 laddr=10.244.3.221:6379 fd=36 name= age=81311 idle=218 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=53 addr=10.244.3.222:50984 laddr=10.244.3.221:6379 fd=37 name= age=81311 idle=288 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16383 oll=98 omem=2009392 tot-mem=2029896 events=rw cmd=unsubscribe user=default redir=-1
id=44 addr=10.244.2.144:45748 laddr=10.244.3.221:6379 fd=28 name= age=81312 idle=316 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=238 addr=10.244.4.16:60572 laddr=10.244.3.221:6379 fd=218 name= age=81299 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=19798 addr=10.244.1.33:57504 laddr=10.244.3.221:6379 fd=339 name= age=5643 idle=5621 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=334 addr=10.244.4.16:33574 laddr=10.244.3.221:6379 fd=303 name= age=81248 idle=17 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=45 addr=10.244.4.16:59950 laddr=10.244.3.221:6379 fd=29 name= age=81312 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=222 addr=10.244.2.144:46234 laddr=10.244.3.221:6379 fd=204 name= age=81301 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=386 addr=10.244.2.144:52874 laddr=10.244.3.221:6379 fd=305 name= age=81007 idle=1745 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16383 oll=16 omem=328064 tot-mem=348568 events=rw cmd=unsubscribe user=default redir=-1
id=40 addr=10.244.4.17:37640 laddr=10.244.3.221:6379 fd=24 name= age=81312 idle=24 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=95 addr=10.244.2.144:45830 laddr=10.244.3.221:6379 fd=78 name= age=81309 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=20006 addr=10.244.1.33:56574 laddr=10.244.3.221:6379 fd=349 name= age=5009 idle=5009 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=103 addr=10.244.2.145:54378 laddr=10.244.3.221:6379 fd=86 name= age=81309 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=305 addr=10.244.2.145:55088 laddr=10.244.3.221:6379 fd=284 name= age=81294 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=287 addr=10.244.1.38:60718 laddr=10.244.3.221:6379 fd=266 name= age=81297 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=146 addr=10.244.1.38:58510 laddr=10.244.3.221:6379 fd=128 name= age=81307 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16348 oll=1858 omem=38096432 tot-mem=38116936 events=rw cmd=unsubscribe user=default redir=-1
id=104 addr=10.244.2.145:54380 laddr=10.244.3.221:6379 fd=87 name= age=81309 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=209 addr=10.244.2.144:46146 laddr=10.244.3.221:6379 fd=191 name= age=81302 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=210 addr=10.244.1.38:59664 laddr=10.244.3.221:6379 fd=192 name= age=81302 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=14435 addr=10.244.1.33:32812 laddr=10.244.3.221:6379 fd=326 name= age=26473 idle=5390 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=211 addr=10.244.3.222:34248 laddr=10.244.3.221:6379 fd=193 name= age=81302 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=212 addr=10.244.3.222:34264 laddr=10.244.3.221:6379 fd=194 name= age=81302 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=226 addr=10.244.2.145:54786 laddr=10.244.3.221:6379 fd=207 name= age=81300 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=327670 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=348184 events=r cmd=rpush user=default redir=-1
id=227 addr=10.244.2.145:54788 laddr=10.244.3.221:6379 fd=208 name= age=81300 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=295 addr=10.244.2.144:46466 laddr=10.244.3.221:6379 fd=274 name= age=81296 idle=6398 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=213 addr=10.244.3.222:34272 laddr=10.244.3.221:6379 fd=195 name= age=81302 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=311 addr=10.244.2.145:55122 laddr=10.244.3.221:6379 fd=292 name= age=81292 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=262 addr=10.244.4.16:60630 laddr=10.244.3.221:6379 fd=241 name= age=81298 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=19923 addr=10.244.2.145:45636 laddr=10.244.3.221:6379 fd=348 name= age=5250 idle=3842 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=214 addr=10.244.3.222:34286 laddr=10.244.3.221:6379 fd=196 name= age=81302 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=215 addr=10.244.2.144:46148 laddr=10.244.3.221:6379 fd=197 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=229366 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=249872 events=r cmd=get user=default redir=-1
id=216 addr=10.244.2.144:46150 laddr=10.244.3.221:6379 fd=198 name= age=81302 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=163 addr=10.244.2.145:54588 laddr=10.244.3.221:6379 fd=145 name= age=81306 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=373 addr=10.244.1.38:52116 laddr=10.244.3.221:6379 fd=304 name= age=81068 idle=56 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=164 addr=10.244.4.17:37968 laddr=10.244.3.221:6379 fd=146 name= age=81306 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=524278 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=544800 events=r cmd=lrange user=default redir=-1
id=113 addr=10.244.4.17:37782 laddr=10.244.3.221:6379 fd=96 name= age=81309 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=20005 addr=10.244.1.33:55958 laddr=10.244.3.221:6379 fd=333 name= age=5011 idle=5011 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default redir=-1
id=114 addr=10.244.1.38:57910 laddr=10.244.3.221:6379 fd=97 name= age=81309 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=115 addr=10.244.3.222:51102 laddr=10.244.3.221:6379 fd=98 name= age=81309 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=331 addr=10.244.3.223:33286 laddr=10.244.3.221:6379 fd=302 name= age=81254 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=14439 addr=10.244.1.33:33150 laddr=10.244.3.221:6379 fd=329 name= age=26471 idle=5390 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=168 addr=10.244.2.145:54634 laddr=10.244.3.221:6379 fd=150 name= age=81304 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1
id=116 addr=10.244.2.145:54410 laddr=10.244.3.221:6379 fd=99 name= age=81309 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=98294 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=118800 events=r cmd=get user=default redir=-1
id=117 addr=10.244.2.144:45878 laddr=10.244.3.221:6379 fd=100 name= age=81309 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=105 addr=10.244.3.222:51092 laddr=10.244.3.221:6379 fd=88 name= age=81309 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=655350 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=675864 events=r cmd=rpush user=default redir=-1
id=284 addr=10.244.4.17:38366 laddr=10.244.3.221:6379 fd=263 name= age=81297 idle=277 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16348 oll=111 omem=2275944 tot-mem=2296448 events=rw cmd=unsubscribe user=default redir=-1
id=285 addr=10.244.2.144:46404 laddr=10.244.3.221:6379 fd=264 name= age=81297 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1
id=286 addr=10.244.2.145:54940 laddr=10.244.3.221:6379 fd=265 name= age=81297 idle=1906 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16354 oll=25 omem=512600 tot-mem=533104 events=rw cmd=unsubscribe user=default redir=-1
id=19870 addr=10.244.1.33:33678 laddr=10.244.3.221:6379 fd=338 name= age=5396 idle=5393 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=zrem user=default redir=-1
id=106 addr=10.244.2.145:54382 laddr=10.244.3.221:6379 fd=89 name= age=81309 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=107 addr=10.244.4.16:60064 laddr=10.244.3.221:6379 fd=90 name= age=81309 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=917494 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=938000 events=r cmd=get user=default redir=-1
id=202 addr=10.244.2.144:46140 laddr=10.244.3.221:6379 fd=184 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=229376 qbuf-free=425974 argv-mem=50 obl=0 oll=0 omem=0 tot-mem=675914 events=r cmd=get user=default redir=-1
id=14434 addr=10.244.1.33:32798 laddr=10.244.3.221:6379 fd=325 name= age=26473 idle=5310 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=203 addr=10.244.4.17:38104 laddr=10.244.3.221:6379 fd=185 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=458742 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=479256 events=r cmd=rpush user=default redir=-1
id=204 addr=10.244.3.222:34192 laddr=10.244.3.221:6379 fd=186 name= age=81302 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=225 addr=10.244.2.145:54784 laddr=10.244.3.221:6379 fd=206 name= age=81300 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=205 addr=10.244.3.222:34198 laddr=10.244.3.221:6379 fd=187 name= age=81302 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=206 addr=10.244.3.222:34208 laddr=10.244.3.221:6379 fd=188 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=262134 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=282640 events=r cmd=get user=default redir=-1
id=207 addr=10.244.3.222:34222 laddr=10.244.3.221:6379 fd=189 name= age=81302 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=308 addr=10.244.2.144:46566 laddr=10.244.3.221:6379 fd=289 name= age=81293 idle=59106 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=21153 addr=10.244.1.33:42858 laddr=10.244.3.221:6379 fd=327 name= age=593 idle=593 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=exists user=default redir=-1
id=260 addr=10.244.4.17:38322 laddr=10.244.3.221:6379 fd=239 name= age=81298 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=310 addr=10.244.4.16:60818 laddr=10.244.3.221:6379 fd=291 name= age=81292 idle=25577 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=261 addr=10.244.3.222:52854 laddr=10.244.3.221:6379 fd=240 name= age=81298 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=208 addr=10.244.3.222:34232 laddr=10.244.3.221:6379 fd=190 name= age=81302 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=108 addr=10.244.4.17:37766 laddr=10.244.3.221:6379 fd=91 name= age=81309 idle=1 flags=P db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16333 oll=9549 omem=195792696 tot-mem=195813200 events=rw cmd=subscribe user=default redir=-1
id=144 addr=10.244.2.144:45992 laddr=10.244.3.221:6379 fd=126 name= age=81307 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=145 addr=10.244.2.144:45998 laddr=10.244.3.221:6379 fd=127 name= age=81307 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=188 addr=10.244.2.144:46138 laddr=10.244.3.221:6379 fd=170 name= age=81302 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=49146 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=69648 events=r cmd=get user=default redir=-1
id=189 addr=10.244.4.17:38070 laddr=10.244.3.221:6379 fd=171 name= age=81302 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=917494 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=938000 events=r cmd=get user=default redir=-1
id=190 addr=10.244.4.17:38072 laddr=10.244.3.221:6379 fd=172 name= age=81302 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=318 addr=10.244.4.16:60852 laddr=10.244.3.221:6379 fd=298 name= age=81291 idle=97 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=191 addr=10.244.1.38:59472 laddr=10.244.3.221:6379 fd=173 name= age=81302 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=192 addr=10.244.4.17:38074 laddr=10.244.3.221:6379 fd=174 name= age=81302 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=193 addr=10.244.1.38:59474 laddr=10.244.3.221:6379 fd=175 name= age=81302 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=443 addr=10.244.4.16:44744 laddr=10.244.3.221:6379 fd=309 name= age=80749 idle=417 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=194 addr=10.244.1.38:59480 laddr=10.244.3.221:6379 fd=176 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=327670 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=348176 events=r cmd=get user=default redir=-1
id=195 addr=10.244.1.38:59482 laddr=10.244.3.221:6379 fd=177 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=163830 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=184336 events=r cmd=get user=default redir=-1
id=196 addr=10.244.4.17:38084 laddr=10.244.3.221:6379 fd=178 name= age=81302 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=328 addr=10.244.2.144:47196 laddr=10.244.3.221:6379 fd=301 name= age=81267 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=197 addr=10.244.4.17:38086 laddr=10.244.3.221:6379 fd=179 name= age=81302 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=198 addr=10.244.1.38:59494 laddr=10.244.3.221:6379 fd=180 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=917494 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=938000 events=r cmd=get user=default redir=-1
id=304 addr=10.244.2.145:55084 laddr=10.244.3.221:6379 fd=283 name= age=81294 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=19990 addr=10.244.1.33:45288 laddr=10.244.3.221:6379 fd=351 name= age=5051 idle=4977 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=scan user=default redir=-1
id=199 addr=10.244.4.16:60388 laddr=10.244.3.221:6379 fd=181 name= age=81302 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=917494 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=938008 events=r cmd=rpush user=default redir=-1
id=200 addr=10.244.4.17:38092 laddr=10.244.3.221:6379 fd=182 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=327670 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=348176 events=r cmd=get user=default redir=-1
id=201 addr=10.244.3.222:34188 laddr=10.244.3.221:6379 fd=183 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=327670 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=348176 events=r cmd=get user=default redir=-1
id=109 addr=10.244.2.144:45854 laddr=10.244.3.221:6379 fd=92 name= age=81309 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=655350 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=675856 events=r cmd=get user=default redir=-1
id=162 addr=10.244.2.144:46048 laddr=10.244.3.221:6379 fd=144 name= age=81306 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=110 addr=10.244.1.38:57862 laddr=10.244.3.221:6379 fd=93 name= age=81309 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=258 addr=10.244.3.222:52834 laddr=10.244.3.221:6379 fd=237 name= age=81298 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=259 addr=10.244.3.222:52848 laddr=10.244.3.221:6379 fd=238 name= age=81298 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=294 addr=10.244.2.144:46452 laddr=10.244.3.221:6379 fd=273 name= age=81296 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=1572854 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=1593376 events=r cmd=lrange user=default redir=-1
id=2802 addr=10.244.1.38:42816 laddr=10.244.3.221:6379 fd=316 name= age=71538 idle=3476 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=unsubscribe user=default redir=-1
id=122 addr=10.244.2.145:54478 laddr=10.244.3.221:6379 fd=104 name= age=81308 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=16382 oll=2736 omem=56098944 tot-mem=56119448 events=rw cmd=unsubscribe user=default redir=-1
id=309 addr=10.244.2.145:55112 laddr=10.244.3.221:6379 fd=290 name= age=81292 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1
id=111 addr=10.244.1.38:57876 laddr=10.244.3.221:6379 fd=94 name= age=81309 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=177 addr=10.244.2.144:46130 laddr=10.244.3.221:6379 fd=159 name= age=81303 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=655350 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=675856 events=r cmd=get user=default redir=-1
id=178 addr=10.244.2.144:46134 laddr=10.244.3.221:6379 fd=160 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=98294 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=118808 events=r cmd=rpush user=default redir=-1
id=179 addr=10.244.4.17:38052 laddr=10.244.3.221:6379 fd=161 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=262134 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=282640 events=r cmd=get user=default redir=-1
id=19747 addr=10.244.1.33:45128 laddr=10.244.3.221:6379 fd=334 name= age=5776 idle=3842 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=180 addr=10.244.1.38:59436 laddr=10.244.3.221:6379 fd=162 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=163830 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=184336 events=r cmd=get user=default redir=-1
id=237 addr=10.244.4.17:38256 laddr=10.244.3.221:6379 fd=217 name= age=81299 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1
id=181 addr=10.244.1.38:59438 laddr=10.244.3.221:6379 fd=163 name= age=81302 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=182 addr=10.244.4.17:38056 laddr=10.244.3.221:6379 fd=164 name= age=81302 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=402 addr=10.244.4.17:46126 laddr=10.244.3.221:6379 fd=311 name= age=80947 idle=185 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=183 addr=10.244.4.17:38058 laddr=10.244.3.221:6379 fd=165 name= age=81302 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=98294 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=118800 events=r cmd=get user=default redir=-1
id=184 addr=10.244.4.16:60362 laddr=10.244.3.221:6379 fd=166 name= age=81302 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=185 addr=10.244.1.38:59444 laddr=10.244.3.221:6379 fd=167 name= age=81302 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=186 addr=10.244.1.38:59446 laddr=10.244.3.221:6379 fd=168 name= age=81302 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=281 addr=10.244.2.144:46394 laddr=10.244.3.221:6379 fd=260 name= age=81297 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=282 addr=10.244.4.17:38358 laddr=10.244.3.221:6379 fd=261 name= age=81297 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1
id=283 addr=10.244.2.144:46398 laddr=10.244.3.221:6379 fd=262 name= age=81297 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=19759 addr=10.244.1.33:50826 laddr=10.244.3.221:6379 fd=337 name= age=5755 idle=3842 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=187 addr=10.244.4.16:60364 laddr=10.244.3.221:6379 fd=169 name= age=81302 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=112 addr=10.244.4.17:37780 laddr=10.244.3.221:6379 fd=95 name= age=81309 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=96 addr=10.244.1.38:57784 laddr=10.244.3.221:6379 fd=79 name= age=81309 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=97 addr=10.244.4.16:60040 laddr=10.244.3.221:6379 fd=80 name= age=81309 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=327670 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=348184 events=r cmd=rpush user=default redir=-1
id=98 addr=10.244.4.17:37742 laddr=10.244.3.221:6379 fd=81 name= age=81309 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=327670 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=348176 events=r cmd=get user=default redir=-1
id=99 addr=10.244.2.144:45842 laddr=10.244.3.221:6379 fd=82 name= age=81309 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=255 addr=10.244.4.17:38304 laddr=10.244.3.221:6379 fd=234 name= age=81298 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=256 addr=10.244.4.17:38306 laddr=10.244.3.221:6379 fd=235 name= age=81298 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1
id=292 addr=10.244.2.145:54980 laddr=10.244.3.221:6379 fd=271 name= age=81296 idle=24217 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=unsubscribe user=default redir=-1
id=293 addr=10.244.2.145:54982 laddr=10.244.3.221:6379 fd=272 name= age=81296 idle=1906 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=20308 addr=10.244.1.33:59672 laddr=10.244.3.221:6379 fd=324 name= age=3884 idle=3841 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=zrem user=default redir=-1
id=20309 addr=10.244.1.33:59674 laddr=10.244.3.221:6379 fd=353 name= age=3884 idle=3171 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=zrem user=default redir=-1
id=257 addr=10.244.4.16:60608 laddr=10.244.3.221:6379 fd=236 name= age=81298 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=100 addr=10.244.4.17:37752 laddr=10.244.3.221:6379 fd=83 name= age=81309 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=get user=default redir=-1
id=143 addr=10.244.3.222:34144 laddr=10.244.3.221:6379 fd=125 name= age=81307 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=159 addr=10.244.1.38:58874 laddr=10.244.3.221:6379 fd=141 name= age=81306 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=160 addr=10.244.1.38:58880 laddr=10.244.3.221:6379 fd=142 name= age=81306 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=393206 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=413728 events=r cmd=lrange user=default redir=-1
id=20016 addr=10.244.1.33:35590 laddr=10.244.3.221:6379 fd=350 name= age=4979 idle=4977 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=zrem user=default redir=-1
id=20018 addr=10.244.1.33:35596 laddr=10.244.3.221:6379 fd=354 name= age=4979 idle=4978 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=zrem user=default redir=-1
id=161 addr=10.244.2.145:54576 laddr=10.244.3.221:6379 fd=143 name= age=81306 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=101 addr=10.244.4.16:60056 laddr=10.244.3.221:6379 fd=84 name= age=81309 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=19804 addr=10.244.1.33:34010 laddr=10.244.3.221:6379 fd=328 name= age=5623 idle=5621 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=zrem user=default redir=-1
id=221 addr=10.244.4.16:60496 laddr=10.244.3.221:6379 fd=203 name= age=81301 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=236 addr=10.244.2.145:54848 laddr=10.244.3.221:6379 fd=215 name= age=81299 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=49146 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=69656 events=r cmd=rpush user=default redir=-1
id=102 addr=10.244.3.222:51082 laddr=10.244.3.221:6379 fd=85 name= age=81309 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=262134 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=282640 events=r cmd=get user=default redir=-1
id=358 addr=10.244.2.145:58544 laddr=10.244.3.221:6379 fd=9 name= age=81138 idle=57 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=174 addr=10.244.4.16:60336 laddr=10.244.3.221:6379 fd=156 name= age=81303 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=262134 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=282640 events=r cmd=get user=default redir=-1
id=175 addr=10.244.4.16:60344 laddr=10.244.3.221:6379 fd=157 name= age=81303 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=229366 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=249872 events=r cmd=get user=default redir=-1
id=19875 addr=10.244.2.144:33480 laddr=10.244.3.221:6379 fd=343 name= age=5391 idle=3947 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=176 addr=10.244.4.16:60346 laddr=10.244.3.221:6379 fd=158 name= age=81303 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=262134 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=282640 events=r cmd=get user=default redir=-1
id=25 addr=10.244.2.144:45520 laddr=10.244.3.221:6379 fd=15 name= age=81337 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61448 events=r cmd=exec user=default redir=-1
id=51 addr=10.244.3.222:50978 laddr=10.244.3.221:6379 fd=35 name= age=81311 idle=1608 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=del user=default redir=-1
id=279 addr=10.244.1.38:60650 laddr=10.244.3.221:6379 fd=258 name= age=81297 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=344 addr=10.244.3.222:60786 laddr=10.244.3.221:6379 fd=20 name= age=81200 idle=49 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=280 addr=10.244.3.222:52900 laddr=10.244.3.221:6379 fd=259 name= age=81297 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=927 addr=10.244.3.222:35508 laddr=10.244.3.221:6379 fd=313 name= age=78778 idle=6996 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=303 addr=10.244.2.144:46542 laddr=10.244.3.221:6379 fd=282 name= age=81294 idle=59106 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=unsubscribe user=default redir=-1
id=41 addr=10.244.1.38:57576 laddr=10.244.3.221:6379 fd=25 name= age=81312 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=92 addr=10.244.4.17:37728 laddr=10.244.3.221:6379 fd=75 name= age=81309 idle=1277 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=93 addr=10.244.4.17:37732 laddr=10.244.3.221:6379 fd=76 name= age=81309 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=163830 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=184336 events=r cmd=get user=default redir=-1
id=94 addr=10.244.2.144:45826 laddr=10.244.3.221:6379 fd=77 name= age=81309 idle=48 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=141 addr=10.244.3.223:42146 laddr=10.244.3.221:6379 fd=123 name= age=81307 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=253 addr=10.244.2.144:46338 laddr=10.244.3.221:6379 fd=232 name= age=81298 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=254 addr=10.244.4.16:60602 laddr=10.244.3.221:6379 fd=233 name= age=81298 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=142 addr=10.244.4.17:37890 laddr=10.244.3.221:6379 fd=124 name= age=81307 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=157 addr=10.244.4.16:60254 laddr=10.244.3.221:6379 fd=139 name= age=81306 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=2621430 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=2641952 events=r cmd=lrange user=default redir=-1
id=321 addr=10.244.1.38:35496 laddr=10.244.3.221:6379 fd=300 name= age=81285 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=lrange user=default redir=-1
id=158 addr=10.244.4.17:37956 laddr=10.244.3.221:6379 fd=140 name= age=81306 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=56 addr=10.244.4.16:59976 laddr=10.244.3.221:6379 fd=39 name= age=81311 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=172 addr=10.244.4.16:60332 laddr=10.244.3.221:6379 fd=154 name= age=81303 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=173 addr=10.244.4.16:60334 laddr=10.244.3.221:6379 fd=155 name= age=81303 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=917494 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=938008 events=r cmd=rpush user=default redir=-1
id=87 addr=10.244.4.17:37714 laddr=10.244.3.221:6379 fd=70 name= age=81309 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=262134 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=282640 events=r cmd=get user=default redir=-1
id=88 addr=10.244.2.144:45816 laddr=10.244.3.221:6379 fd=71 name= age=81309 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=89 addr=10.244.2.144:45818 laddr=10.244.3.221:6379 fd=72 name= age=81309 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=229366 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=249872 events=r cmd=get user=default redir=-1
id=90 addr=10.244.4.16:60016 laddr=10.244.3.221:6379 fd=73 name= age=81309 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=21306 addr=10.244.3.235:45710 laddr=10.244.3.221:6379 fd=352 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=26 qbuf-free=40928 argv-mem=10 obl=0 oll=0 omem=0 tot-mem=61466 events=r cmd=client user=default redir=-1
Comment From: mspi92
I wasn't aware of clients.normal being a memory size, looked to me like a "count". I now see that there are some clients which have a tot-mem value of around 180MB with command subscribe/unsubscribe. What exactly causes the tot-mem value? Is it an application design issue/am i missing an important information that connections need to be closed after time whatever? It is very hard to find any issue because basically any bug report is including the word "memory" and "overhead" because of the redis info :D
Comment From: sundb
the reason maybe is that the sub clents cant keep up with pub.
Comment From: mspi92
Ok, with your help i was able to pinpoint a) where exactly the memory was consumed (which was in the client output buffers) and b) what consumes it and with that i was able to locate the real issue:
https://github.com/celery/celery/issues/6963
Thank you very much.