Redis Version: 7.2.0 OS: Rocky Linux release 8.8 Redis Mode: Standalone Configuration: 5 nodes with sentinel running on each

Redis was upgraded from 7.0.12 to 7.2.0 last month. From last week, nodes are going down continuously. We are seeing master node down message in sentinel logs:

766629:X 27 Oct 2023 05:24:44.668 # +sdown master mymaster x.x.x.128 6380 766629:X 27 Oct 2023 05:24:44.731 # +odown master mymaster x.x.x.128 6380 #quorum 4/2 766629:X 27 Oct 2023 05:25:10.625 # +sdown master mymaster x.x.x.130 6380 766629:X 27 Oct 2023 05:25:11.715 # +odown master mymaster x.x.x.130 6380 #quorum 5/2 766629:X 27 Oct 2023 05:25:49.184 # +sdown master mymaster x.x.x.129 6380 766629:X 27 Oct 2023 05:25:50.269 # +odown master mymaster x.x.x.129 6380 #quorum 4/2 766629:X 27 Oct 2023 05:26:16.488 # +sdown master mymaster x.x.x.131 6380 766629:X 27 Oct 2023 05:26:16.546 # +odown master mymaster x.x.x.131 6380 #quorum 3/2

For node ending with 128 IP, only found below message in log when it went down:

765694:M 27 Oct 2023 05:20:20.384 * Background saving terminated with success 765694:M 27 Oct 2023 05:23:36.606 * 10000 changes in 60 seconds. Saving... 765694:M 27 Oct 2023 05:23:36.825 * Background saving started by pid 1460646 765694:M 27 Oct 2023 05:24:01.518 # Can't fork for module: File exists 1460646:C 27 Oct 2023 05:24:03.862 * DB saved on disk 1460646:C 27 Oct 2023 05:24:04.127 * Fork CoW for RDB: current 179 MB, peak 179 MB, average 131 MB 765694:M 27 Oct 2023 05:24:04.420 # Can't fork for module: File exists 765694:M 27 Oct 2023 05:24:04.427 * Background saving terminated with success 765694:signal-handler (1698385061) Received SIGTERM scheduling shutdown...

CPU usage is higher than usual and memory usage is around 8-9 GB. Both of them are not reaching any limits. Resources are available.

As part of troubleshooting, we have tried re-installing Redis on each node but still issue persists.

Redis.conf:

bind 0.0.0.0 port 6380 daemonize yes loadmodule /home/redis/redis_sentinel_dev/data/modules/redistimeseries.so loadmodule /home/redis/redis_sentinel_dev/data/modules/redisearch.so TIMEOUT 0 MAXSEARCHRESULTS -1 MINPREFIX 2 loadmodule /home/redis/redis_sentinel_dev/data/modules/rejson.so loadmodule /home/redis/redis_sentinel_dev/data/modules/libredisgears.so v8-plugin-path /home/redis/redis_sentinel_dev/data/modules/libredisgears_v8_plugin.so pidfile "/home/redis/redis_sentinel_dev/data/run/redis.pid" logfile "/home/redis/redis_sentinel_dev/data/log/redis.log" dir "/data/redis/dev" save 300 100 save 60 10000 latency-tracking-info-percentiles 50 99 99.9

I have attached gdb trace and waiting for next crash. What else can be tried to find root cause and fix?

Comment From: rishinair19

This is what I got in gdb stack trace:

[Detaching after fork from child process 2611295] bt ^C Thread 1 "redis-server" received signal SIGINT, Interrupt. 0x00007ffff6160fd1 in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so (gdb) bt

0 0x00007ffff6160fd1 in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

1 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

2 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

3 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

4 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

5 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

6 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

7 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

8 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

9 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

10 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

11 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

12 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

13 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

14 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

15 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

16 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

17 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

18 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

19 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

20 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

21 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

22 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

23 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

24 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

25 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

26 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

27 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

28 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

29 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

30 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

31 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

32 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

33 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

34 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

35 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

36 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

37 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

38 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

39 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

40 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

41 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

42 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

43 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

44 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

45 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

46 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

47 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

48 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

49 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

50 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

51 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

52 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

53 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

54 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

55 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

56 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

57 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

58 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

59 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

60 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

61 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

62 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

63 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

64 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

65 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

66 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

67 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

--Type for more, q to quit, c to continue without paging--c

68 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

69 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

70 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

71 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

72 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

73 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

74 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

75 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

76 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

77 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

78 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

79 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

80 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

81 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

82 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

83 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

84 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

85 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

86 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

87 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

88 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

89 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

90 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

91 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

92 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

93 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

94 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

95 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

96 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

97 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

98 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

99 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

100 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

101 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

102 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

103 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

104 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

105 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

106 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

107 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

108 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

109 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

110 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

111 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

112 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

113 0x00007ffff616119c in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

114 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

115 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

116 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

117 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

118 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

119 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

120 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

121 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

122 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

123 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

124 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

125 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

126 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

127 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

128 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

129 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

130 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

131 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

132 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

133 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

134 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

135 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

136 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

137 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

138 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

139 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

140 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

141 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

142 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

143 0x00007ffff6161090 in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

144 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

145 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

146 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

147 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

148 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

149 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

150 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

151 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

152 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

153 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

154 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

155 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

156 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

157 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

158 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

159 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

160 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

161 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

162 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

163 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

164 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

165 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

166 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

167 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

168 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

169 0x00007ffff6160fce in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

170 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

171 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

172 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

173 0x00007ffff616119c in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

174 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

175 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

176 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

177 0x00007ffff616119c in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

178 0x00007ffff6161090 in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

179 0x00007ffff61611ec in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

180 0x00007ffff6161090 in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

181 0x00007ffff616119c in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

182 0x00007ffff6161090 in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

183 0x00007ffff6161090 in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

184 0x00007ffff6161090 in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

185 0x00007ffff616101c in containsIterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

186 0x00007ffff6163448 in TrieNode_IterateContains () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

187 0x00007ffff61377b8 in Query_EvalNode () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

188 0x00007ffff613713c in Query_EvalNode () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

189 0x00007ffff613713c in Query_EvalNode () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

190 0x00007ffff613713c in Query_EvalNode () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

191 0x00007ffff613713c in Query_EvalNode () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

192 0x00007ffff61371d4 in Query_EvalNode () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

193 0x00007ffff6137e36 in QAST_Iterate () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

194 0x00007ffff60e4c7a in prepareExecutionPlan () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

195 0x00007ffff60e56fc in RSSearchCommand () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

196 0x00000000004a44d3 in RedisModuleCommandDispatcher (c=0x7ffea6661c00) at module.c:917

197 0x0000000000557be0 in call (c=c@entry=0x7ffea6661c00, flags=flags@entry=3) at server.c:3519

198 0x0000000000558c24 in processCommand (c=0x7ffea6661c00) at server.c:4160

199 0x000000000053ad59 in processCommandAndResetClient (c=0x7ffea6661c00) at networking.c:2574

200 processInputBuffer (c=c@entry=0x7ffea6661c00) at networking.c:2574

201 0x000000000053b288 in readQueryFromClient (conn=0x7fff15244100) at networking.c:2713

202 0x000000000045f3cc in callHandler (handler=, conn=0x7fff15244100) at connhelpers.h:79

203 connSocketEventHandler (el=, fd=, clientData=0x7fff15244100, mask=) at socket.c:298

204 0x0000000000561299 in aeProcessEvents (flags=27, eventLoop=0x7ffff6c2a1e0) at ae.c:276

205 aeMain (eventLoop=0x7ffff6c2a1e0) at ae.c:496

206 0x000000000044f86a in main (argc=, argv=0x7fffffffecb8) at server.c:7360

Comment From: rishinair19

I had compiled redissearch.so locally as GNU libc version was 2.28 here. Is there something wrong with it?

Comment From: rishinair19

From another crash:

Thread 1 "redis-server" received signal SIGINT, Interrupt. 0x00007ffff6165baf in dictFind () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so (gdb) bt

0 0x00007ffff6165baf in dictFind () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

1 0x00007ffff6165c49 in dictFetchValue () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

2 0x00007ffff6145643 in Redis_OpenInvertedIndexEx () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

3 0x00007ffff6121d86 in Indexer_Process () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

4 0x00007ffff6121f9d in Indexer_Add () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

5 0x00007ffff610a5ad in Document_AddToIndexes () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

6 0x00007ffff6157470 in IndexSpec_UpdateDoc () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

7 0x00007ffff6157be2 in Indexes_UpdateMatchingWithSchemaRules () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

8 0x00007ffff612dcd4 in HashNotificationCallback () from /home/redis/redis_sentinel_dev/data/modules/redisearch.so

9 0x000000000048c4e4 in moduleNotifyKeyspaceEvent (type=4096, event=0x644ca8 "loaded", key=0x7fffffffa3c0, dbid=0) at module.c:8724

10 0x0000000000519081 in moduleNotifyKeyspaceEvent (dbid=, key=0x7fffffffa3c0, event=0x644ca8 "loaded", type=4096) at module.c:8681

11 rdbLoadRioWithLoadingCtx (rdb=rdb@entry=0x7fffffffa860, rdbflags=rdbflags@entry=2, rsi=rsi@entry=0x7fffffffa9e0, rdb_loading_ctx=rdb_loading_ctx@entry=0x7fffffffa850) at rdb.c:3313

12 0x0000000000519d3e in rdbLoadRio (rsi=0x7fffffffa9e0, rdbflags=2, rdb=0x7fffffffa860) at functions.c:218

13 rdbLoad (filename=0x7ffff6c15100 "dump.rdb", rsi=0x7fffffffa9e0, rdbflags=2) at rdb.c:3397

14 0x00000000005249a9 in readSyncBulkPayload (conn=) at replication.c:2228

15 0x000000000045f3cc in callHandler (handler=, conn=0x7ffddf89ea40) at connhelpers.h:79

16 connSocketEventHandler (el=, fd=, clientData=0x7ffddf89ea40, mask=) at socket.c:298

17 0x0000000000561299 in aeProcessEvents (flags=27, eventLoop=0x7ffff6c2a1e0) at ae.c:276

18 aeMain (eventLoop=0x7ffff6c2a1e0) at ae.c:496

19 0x000000000044f86a in main (argc=, argv=0x7fffffffecb8) at server.c:7360

Comment From: zuiderkwast

SIGTERM is not a crash from within redis. It is a signal from another program on the same machine telling redis to shutdown.

^C Thread 1 "redis-server" received signal SIGINT, Interrupt.

SIGINT is also a signal that other programs can send to redis and which causes redis to shutdown. Normally it is sent when you press Ctrl-C when redis is running in the foreground. ^C indicates that Ctrl-C was pressed. Was it?

Regarding the modules like redisearch, I don't know much about them, but there is a separate repository for it here: https://github.com/RediSearch/RediSearch

Comment From: rishinair19

Checked for any external factors that could send SIGTERM but there weren't any. And you're see SIGINT in trace as I had pressed Ctrl + C to stop the trace after Redis stopped responding.

Another thing I observed is that when nodes are about to go down, CPU is lot higher than usual. Its usually between 5-10% but at time of issue it goes to 100% (1 core).

Comment From: zuiderkwast

Just a guess: Maybe there is a program in your system that automatically shuts down programs that consume 100% CPU by sending SIGTERM to them? It would explain the issue.

If not, I don't know what can cause this.

Comment From: rishinair19

We thought it might be the OS, so we moved Redis to freshly installed machine but the issues were there. We had disabled timeout in redis.conf a few months back so thought to revert that change.

loadmodule /home/redis/redis_sentinel_dev/data/modules/redisearch.so TIMEOUT 0 MAXSEARCHRESULTS -1 MINPREFIX 2 We reduced the timeout to 30 seconds and so far haven't had this issue.

Comment From: rishinair19

Closing since issue is now resolved