environment
[root@localhost ~]# redis-cli version
Could not connect to Redis at 127.0.0.1:6379: Connection refused
[root@localhost ~]# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/opt/rh/devtoolset-9/root/usr --mandir=/opt/rh/devtoolset-9/root/usr/share/man --infodir=/opt/rh/devtoolset-9/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --with-default-libstdcxx-abi=gcc4-compatible --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-9.3.1-20200408/obj-x86_64-redhat-linux/isl-install --disable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC)
[root@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)
[root@localhost ~]#
what i do
create a cluster
redis-cli --cluster create \
--cluster-replicas 1 \
192.168.93.63:7000 \
192.168.93.63:7003 \
192.168.93.64:7001 \
192.168.93.64:7004 \
192.168.93.65:7002 \
192.168.93.65:7005
````
make a backup for the cluster
[root@localhost ~]# redis-cli --cluster backup 192.168.93.63:7000 /opt/redis/backup
Performing Cluster Check (using node 192.168.93.63:7000) M: 58f6728a9929c33c9cdc606e21288f1f4d2eb979 192.168.93.63:7000 slots:[0-5460] (5461 slots) master 1 additional replica(s) M: ddbe0a96c919ee3c805b847c280e5c9ae944a6c1 192.168.93.65:7002 slots:[10923-16383] (5461 slots) master 1 additional replica(s) S: 199558c6620c39181bd1f562b9786aae558a2698 192.168.93.63:7003 slots: (0 slots) slave replicates ddbe0a96c919ee3c805b847c280e5c9ae944a6c1 M: 31850375645392837441f3cf174ffe75797faf0f 192.168.93.64:7001 slots:[5461-10922] (5462 slots) master 1 additional replica(s) S: 971e6a61ddc357af4c942ae64852592b38dc97e7 192.168.93.64:7004 slots: (0 slots) slave replicates 58f6728a9929c33c9cdc606e21288f1f4d2eb979 S: 82356b67e967df84a49435024890a516d9c4c86a 192.168.93.65:7005 slots: (0 slots) slave replicates 31850375645392837441f3cf174ffe75797faf0f [OK] All nodes agree about slots configuration. Check for open slots... Check slots coverage... [OK] All 16384 slots covered. Node 192.168.93.63:7000 -> Saving RDB... SYNC sent to master, writing 175 bytes to '/opt/redis/backup/redis-node-192.168.93.63-7000-58f6728a9929c33c9cdc606e21288f1f4d2eb979.rdb' Transfer finished with success. Transfer finished with success. Node 192.168.93.65:7002 -> Saving RDB... SYNC sent to master, writing 175 bytes to '/opt/redis/backup/redis-node-192.168.93.65-7002-ddbe0a96c919ee3c805b847c280e5c9ae944a6c1.rdb' Transfer finished with success. Transfer finished with success. Node 192.168.93.64:7001 -> Saving RDB... SYNC sent to master, writing 175 bytes to '/opt/redis/backup/redis-node-192.168.93.64-7001-31850375645392837441f3cf174ffe75797faf0f.rdb' Transfer finished with success. Transfer finished with success. Saving cluster configuration to: /opt/redis/backup/nodes.json [OK] Backup created into: /opt/redis/backup *** Error in `redis-cli': free(): invalid pointer: 0x00000000024c24b2 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x81299)[0x7f4a20a22299] redis-cli(redisFree+0x1a)[0x4253da] redis-cli[0x4127f6] redis-cli[0x4129d9] redis-cli(main+0xa34)[0x40d674] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4a209c3555] redis-cli[0x40edd9] ======= Memory map: ======== 00400000-0040c000 r--p 00000000 fd:00 740368 /usr/local/bin/redis-cli 0040c000-0049b000 r-xp 0000c000 fd:00 740368 /usr/local/bin/redis-cli 0049b000-004bf000 r--p 0009b000 fd:00 740368 /usr/local/bin/redis-cli 004c0000-004c2000 r--p 000bf000 fd:00 740368 /usr/local/bin/redis-cli 004c2000-004c6000 rw-p 000c1000 fd:00 740368 /usr/local/bin/redis-cli 004c6000-006d9000 rw-p 00000000 00:00 0 024c2000-024e3000 rw-p 00000000 00:00 0 [heap] 7f4a18000000-7f4a18021000 rw-p 00000000 00:00 0 7f4a18021000-7f4a1c000000 ---p 00000000 00:00 0 7f4a1fdea000-7f4a1fdff000 r-xp 00000000 fd:00 33554508 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f4a1fdff000-7f4a1fffe000 ---p 00015000 fd:00 33554508 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f4a1fffe000-7f4a1ffff000 r--p 00014000 fd:00 33554508 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f4a1ffff000-7f4a20000000 rw-p 00015000 fd:00 33554508 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 7f4a20000000-7f4a20800000 rw-p 00000000 00:00 0 7f4a209a1000-7f4a20b64000 r-xp 00000000 fd:00 33555739 /usr/lib64/libc-2.17.so 7f4a20b64000-7f4a20d64000 ---p 001c3000 fd:00 33555739 /usr/lib64/libc-2.17.so 7f4a20d64000-7f4a20d68000 r--p 001c3000 fd:00 33555739 /usr/lib64/libc-2.17.so 7f4a20d68000-7f4a20d6a000 rw-p 001c7000 fd:00 33555739 /usr/lib64/libc-2.17.so 7f4a20d6a000-7f4a20d6f000 rw-p 00000000 00:00 0 7f4a20d6f000-7f4a20d86000 r-xp 00000000 fd:00 33555765 /usr/lib64/libpthread-2.17.so 7f4a20d86000-7f4a20f85000 ---p 00017000 fd:00 33555765 /usr/lib64/libpthread-2.17.so 7f4a20f85000-7f4a20f86000 r--p 00016000 fd:00 33555765 /usr/lib64/libpthread-2.17.so 7f4a20f86000-7f4a20f87000 rw-p 00017000 fd:00 33555765 /usr/lib64/libpthread-2.17.so 7f4a20f87000-7f4a20f8b000 rw-p 00000000 00:00 0 7f4a20f8b000-7f4a20f92000 r-xp 00000000 fd:00 33555769 /usr/lib64/librt-2.17.so 7f4a20f92000-7f4a21191000 ---p 00007000 fd:00 33555769 /usr/lib64/librt-2.17.so 7f4a21191000-7f4a21192000 r--p 00006000 fd:00 33555769 /usr/lib64/librt-2.17.so 7f4a21192000-7f4a21193000 rw-p 00007000 fd:00 33555769 /usr/lib64/librt-2.17.so 7f4a21193000-7f4a21195000 r-xp 00000000 fd:00 33555745 /usr/lib64/libdl-2.17.so 7f4a21195000-7f4a21395000 ---p 00002000 fd:00 33555745 /usr/lib64/libdl-2.17.so 7f4a21395000-7f4a21396000 r--p 00002000 fd:00 33555745 /usr/lib64/libdl-2.17.so 7f4a21396000-7f4a21397000 rw-p 00003000 fd:00 33555745 /usr/lib64/libdl-2.17.so 7f4a21397000-7f4a21498000 r-xp 00000000 fd:00 33555747 /usr/lib64/libm-2.17.so 7f4a21498000-7f4a21697000 ---p 00101000 fd:00 33555747 /usr/lib64/libm-2.17.so 7f4a21697000-7f4a21698000 r--p 00100000 fd:00 33555747 /usr/lib64/libm-2.17.so 7f4a21698000-7f4a21699000 rw-p 00101000 fd:00 33555747 /usr/lib64/libm-2.17.so 7f4a21699000-7f4a216bb000 r-xp 00000000 fd:00 33555732 /usr/lib64/ld-2.17.so 7f4a218ad000-7f4a218b3000 rw-p 00000000 00:00 0 7f4a218b7000-7f4a218ba000 rw-p 00000000 00:00 0 7f4a218ba000-7f4a218bb000 r--p 00021000 fd:00 33555732 /usr/lib64/ld-2.17.so 7f4a218bb000-7f4a218bc000 rw-p 00022000 fd:00 33555732 /usr/lib64/ld-2.17.so 7f4a218bc000-7f4a218bd000 rw-p 00000000 00:00 0 7ffd27287000-7ffd272a8000 rw-p 00000000 00:00 0 [stack] 7ffd27391000-7ffd27393000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall] Aborted
bakcupfiles
[root@localhost ~]# ls /opt/redis/backup/ nodes.json redis-node-192.168.93.64-7001-31850375645392837441f3cf174ffe75797faf0f.rdb redis-node-192.168.93.63-7000-58f6728a9929c33c9cdc606e21288f1f4d2eb979.rdb redis-node-192.168.93.65-7002-ddbe0a96c919ee3c805b847c280e5c9ae944a6c1.rdb
but there have some error and Aborted at the end
*** Error in `redis-cli': free(): invalid pointer: 0x00000000024c24b2 ***
```
question
so is it success for backup?
Comment From: oranagra
@willzhang it seems that the backup was created successfully (the message [OK] Backup created into is printed when all is done).
can you please tell us which version of redis you are using?
specifically the redis-cli (which might in theory be from a different version).
please do redis-cli --version
Comment From: LelikSan
Hello I have exactly the same issue with backup. redis-cli v 6.0.9; CentOS Linux release 8.1.1911
[OK] Backup created into: /var/backup/ free(): invalid pointer Aborted (core dumped)
Comment From: oranagra
@LelikSan any chance you can share the redis-cli core dump file?
Comment From: LelikSan
coredumpctl gdb -1 PID: 29980 (redis-cli) UID: 0 (root) GID: 0 (root) Signal: 6 (ABRT) Timestamp: Tue 2020-12-29 15:15:28 MSK (2min 30s ago) Command Line: redis-cli --cluster backup 192.168.41.157 6379 /var/backup/ Executable: /usr/bin/redis-cli Control Group: /user.slice/user-786663460.slice/session-222.scope Unit: session-222.scope Slice: user-786663460.slice Session: 222 Owner UID: 786663460 (alexey.filin-3) Boot ID: b21d15450b7d44f485a7434323352ca7 Machine ID: 403c5f4c187f40889a244f1c4e5afa26 Hostname: RDS016 Storage: /var/lib/systemd/coredump/core.redis-cli.0.b21d15450b7d44f485a7434323352ca7.29980.1609244128000000.lz4 Message: Process 29980 (redis-cli) of user 0 dumped core.
Stack trace of thread 29980:
#0 0x00007f4b4eae270f raise (libc.so.6)
#1 0x00007f4b4eaccb25 abort (libc.so.6)
#2 0x00007f4b4eb25897 __libc_message (libc.so.6)
#3 0x00007f4b4eb2bfdc malloc_printerr (libc.so.6)
#4 0x00007f4b4eb2d8dc _int_free (libc.so.6)
#5 0x000055825ac7210e redisFree (redis-cli)
#6 0x000055825ac5d846 freeClusterManagerNode (redis-cli)
#7 0x000055825ac5da79 freeClusterManager (redis-cli)
#8 0x000055825ac57a47 main (redis-cli)
#9 0x00007f4b4eace6a3 __libc_start_main (libc.so.6)
#10 0x000055825ac5968e _start (redis-cli)
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-11.el8 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/redis-cli...Reading symbols from .gnu_debugdata for /usr/bin/redis-cli...(no debugging symbols found)...done. (no debugging symbols found)...done. [New LWP 29980]
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1".
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments
warning: Loadable section ".note.gnu.property" outside of ELF segments Core was generated by `redis-cli --cluster backup 192.168.41.157 6379 /var/backup/'. Program terminated with signal SIGABRT, Aborted.
0 0x00007f4b4eae270f in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: yum debuginfo-install redis-6.0.9-2.el8.remi.x86_64 (gdb)
Comment From: oranagra
turns out this is easy to reproduce an fix (see #8267) and now i also realize it only took one backup and aborted before proceeding to the other cluster nodes.
Comment From: LelikSan
Thank you But backup in the whole was nevertheless successful. It created dumps for three masters and nodes.json file