I am running the redis-benchmark with the following command: /usr/local/bin/redis-benchmark -h 127.0.0.1 -p 10001 -c 50 -n 1000000 -d 10 -k 0 -r 10000 -P 1 -t get
But it always get an error: Error: Connection reset by peer
Here is the config get * output: 127.0.0.1:10001> config get * 1) "dbfilename" 2) "dump.rdb" 3) "requirepass" 4) "" 5) "masterauth" 6) "" 7) "cluster-announce-ip" 8) "" 9) "unixsocket" 10) "" 11) "logfile" 12) "/var/log/redis01.log" 13) "pidfile" 14) "/var/run/redisi01.pid" 15) "slave-announce-ip" 16) "" 17) "maxmemory" 18) "0" 19) "proto-max-bulk-len" 20) "536870912" 21) "client-query-buffer-limit" 22) "1073741824" 23) "maxmemory-samples" 24) "5" 25) "lfu-log-factor" 26) "10" 27) "lfu-decay-time" 28) "1" 29) "timeout" 30) "0" 31) "active-defrag-threshold-lower" 32) "10" 33) "active-defrag-threshold-upper" 34) "100" 35) "active-defrag-ignore-bytes" 36) "104857600" 37) "active-defrag-cycle-min" 38) "25" 39) "active-defrag-cycle-max" 40) "75" 41) "auto-aof-rewrite-percentage" 42) "100" 43) "auto-aof-rewrite-min-size" 44) "67108864" 45) "hash-max-ziplist-entries" 46) "512" 47) "hash-max-ziplist-value" 48) "64" 49) "list-max-ziplist-size" 50) "-2" 51) "list-compress-depth" 52) "0" 53) "set-max-intset-entries" 54) "512" 55) "zset-max-ziplist-entries" 56) "128" 57) "zset-max-ziplist-value" 58) "64" 59) "hll-sparse-max-bytes" 60) "3000" 61) "lua-time-limit" 62) "5000" 63) "slowlog-log-slower-than" 64) "10000" 65) "latency-monitor-threshold" 66) "0" 67) "slowlog-max-len" 68) "128" 69) "port" 70) "10001" 71) "cluster-announce-port" 72) "0" 73) "cluster-announce-bus-port" 74) "0" 75) "tcp-backlog" 76) "65535" 77) "databases" 78) "16" 79) "repl-ping-slave-period" 80) "10" 81) "repl-timeout" 82) "60" 83) "repl-backlog-size" 84) "1048576" 85) "repl-backlog-ttl" 86) "3600" 87) "maxclients" 88) "1000000" 89) "watchdog-period" 90) "0" 91) "slave-priority" 92) "100" 93) "slave-announce-port" 94) "0" 95) "min-slaves-to-write" 96) "0" 97) "min-slaves-max-lag" 98) "10" 99) "hz" 100) "10" 101) "cluster-node-timeout" 102) "15000" 103) "cluster-migration-barrier" 104) "1" 105) "cluster-slave-validity-factor" 106) "10" 107) "repl-diskless-sync-delay" 108) "5" 109) "tcp-keepalive" 110) "0" 111) "cluster-require-full-coverage" 112) "yes" 113) "cluster-slave-no-failover" 114) "no" 115) "no-appendfsync-on-rewrite" 116) "no" 117) "slave-serve-stale-data" 118) "yes" 119) "slave-read-only" 120) "yes" 121) "stop-writes-on-bgsave-error" 122) "yes" 123) "daemonize" 124) "no" 125) "rdbcompression" 126) "yes" 127) "rdbchecksum" 128) "yes" 129) "activerehashing" 130) "yes" 131) "activedefrag" 132) "no" 133) "protected-mode" 134) "no" 135) "repl-disable-tcp-nodelay" 136) "no" 137) "repl-diskless-sync" 138) "no" 139) "aof-rewrite-incremental-fsync" 140) "yes" 141) "aof-load-truncated" 142) "yes" 143) "aof-use-rdb-preamble" 144) "no" 145) "lazyfree-lazy-eviction" 146) "no" 147) "lazyfree-lazy-expire" 148) "no" 149) "lazyfree-lazy-server-del" 150) "no" 151) "slave-lazy-flush" 152) "no" 153) "maxmemory-policy" 154) "noeviction" 155) "loglevel" 156) "verbose" 157) "supervised" 158) "no" 159) "appendfsync" 160) "everysec" 161) "syslog-facility" 162) "local0" 163) "appendonly" 164) "no" 165) "dir" 166) "/var/lib/redis/i01" 167) "save" 168) "" 169) "client-output-buffer-limit" 170) "normal 0 0 0 slave 268435456 67108864 60 pubsub 33554432 8388608 60" 171) "unixsocketperm" 172) "0" 173) "slaveof" 174) "" 175) "notify-keyspace-events" 176) "" 177) "bind" 178) ""
Here is the log file during starting the redis server:
cat /var/log/redis01.log
5464:C 20 Jun 01:30:35.552 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo 5464:C 20 Jun 01:30:35.552 # Redis version=4.0.10, bits=64, commit=00000000, modified=0, pid=5464, just started 5464:C 20 Jun 01:30:35.552 # Configuration loaded 5464:M 20 Jun 01:30:35.553 * Increased maximum number of open files to 1000032 (it was originally set to 65536). 5464:M 20 Jun 01:30:35.557 * Running mode=standalone, port=10001. 5464:M 20 Jun 01:30:35.557 # Server initialized 5464:M 20 Jun 01:30:35.557 * Ready to accept connections 5464:M 20 Jun 01:30:35.558 - 0 clients connected (0 slaves), 58966136 bytes in use 5464:M 20 Jun 01:30:40.563 - 0 clients connected (0 slaves), 58966136 bytes in use
And the ulimit setting:
ulimit -a
core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 3963079 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 65536 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 65536 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Comment From: davidzengxhsh
The redis version is: Redis server v=4.0.10 sha=00000000:0 malloc=jemalloc-4.0.3 bits=64 build=e3e9e1448d49f217
Comment From: filipecosta90
@davidzengxhsh It sounds like you are running out of ephemeral ports, given from the time you start the benchmark until the time you got the error, all connections are in TIME_WAIT state and there is no free connection on the ephemeral port range. I've confirmed it on my machine that by default I get the same behaviour as you (notice the Can't assign requested address):
(base) fco@fcos-Air Desktop % /usr/local/bin/redis-benchmark -h 127.0.0.1 -c 50 -n 1000000 -d 10 -k 0 -r 10000 -P 1 -t get
WARNING: keepalive disabled, you probably need 'echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse' for Linux and 'sudo sysctl -w net.inet.tcp.msl=1000' for Mac OS X in order to use a lot of clients/requests
Could not connect to Redis at 127.0.0.1:6379: Can't assign requested address
and when following up on the recommendation ( and changing from 15000 to 1000 ), meaning the TIME_WAIT interval (2*MSL) changes from 30secs to 2secs, I can the run the benchmarks as usual. Notice is not related to redis performance at all ( or even the opposite, given it can reply very fast, it least a fast exhaustion of the available port range :) ) After the change:
(base) fco@fcos-Air Desktop % sudo sysctl -w net.inet.tcp.msl=1000
Password:
net.inet.tcp.msl: 15000 -> 1000
(base) fco@fcos-Air Desktop % /usr/local/bin/redis-benchmark -h 127.0.0.1 -c 50 -n 1000000 -d 10 -k 1 -r 10000 -P 1 -t get
====== GET ====== )
1000000 requests completed in 10.82 seconds
50 parallel clients
10 bytes payload
keep alive: 1
host configuration "save": 3600 1 300 100 60 10000
host configuration "appendonly": no
multi-thread: no
Latency by percentile distribution:
0.000% <= 0.111 milliseconds (cumulative count 1)
50.000% <= 0.271 milliseconds (cumulative count 573459)
75.000% <= 0.303 milliseconds (cumulative count 759396)
87.500% <= 0.431 milliseconds (cumulative count 877554)
93.750% <= 0.607 milliseconds (cumulative count 937911)
96.875% <= 0.695 milliseconds (cumulative count 969570)
98.438% <= 0.823 milliseconds (cumulative count 984876)
99.219% <= 0.967 milliseconds (cumulative count 992279)
99.609% <= 1.079 milliseconds (cumulative count 996165)
99.805% <= 1.199 milliseconds (cumulative count 998106)
99.902% <= 1.343 milliseconds (cumulative count 999045)
99.951% <= 1.527 milliseconds (cumulative count 999515)
99.976% <= 1.719 milliseconds (cumulative count 999761)
99.988% <= 1.879 milliseconds (cumulative count 999880)
99.994% <= 2.031 milliseconds (cumulative count 999942)
99.997% <= 2.207 milliseconds (cumulative count 999970)
99.998% <= 2.407 milliseconds (cumulative count 999986)
99.999% <= 2.487 milliseconds (cumulative count 999993)
100.000% <= 2.631 milliseconds (cumulative count 999997)
100.000% <= 2.671 milliseconds (cumulative count 999999)
100.000% <= 2.711 milliseconds (cumulative count 1000000)
100.000% <= 2.711 milliseconds (cumulative count 1000000)
Cumulative distribution of latencies:
0.000% <= 0.103 milliseconds (cumulative count 0)
1.287% <= 0.207 milliseconds (cumulative count 12874)
75.940% <= 0.303 milliseconds (cumulative count 759396)
86.855% <= 0.407 milliseconds (cumulative count 868550)
89.889% <= 0.503 milliseconds (cumulative count 898888)
93.791% <= 0.607 milliseconds (cumulative count 937911)
97.101% <= 0.703 milliseconds (cumulative count 971013)
98.371% <= 0.807 milliseconds (cumulative count 983711)
98.958% <= 0.903 milliseconds (cumulative count 989578)
99.376% <= 1.007 milliseconds (cumulative count 993762)
99.674% <= 1.103 milliseconds (cumulative count 996739)
99.818% <= 1.207 milliseconds (cumulative count 998182)
99.887% <= 1.303 milliseconds (cumulative count 998875)
99.925% <= 1.407 milliseconds (cumulative count 999255)
99.947% <= 1.503 milliseconds (cumulative count 999472)
99.963% <= 1.607 milliseconds (cumulative count 999627)
99.974% <= 1.703 milliseconds (cumulative count 999741)
99.984% <= 1.807 milliseconds (cumulative count 999844)
99.989% <= 1.903 milliseconds (cumulative count 999890)
99.993% <= 2.007 milliseconds (cumulative count 999930)
99.996% <= 2.103 milliseconds (cumulative count 999958)
100.000% <= 3.103 milliseconds (cumulative count 1000000)
Summary:
throughput summary: 92387.28 requests per second
latency summary (msec):
avg min p50 p95 p99 max
0.313 0.104 0.271 0.631 0.919 2.711
Closing the issue. feel free to re-open if you can't fix/reproduce on your side :)