- In Redis 6.2.7, the following test case fails often while running
make test OR ./runtest --tls. However, sometimes the test case succeeded without any issues.
When I checked the stdout file STACK TRACE was available. So it seems the STACK TRACE was not available at the time of assertion.
if {!$::valgrind} {
set server_path [tmpdir server1.log]
start_server [list overrides [list dir $server_path]] {
test "Crash report generated on SIGABRT" {
set pid [s process_id]
exec kill -SIGABRT $pid
set pattern "*STACK TRACE*"
set result [exec tail -1000 < [srv 0 stdout]]
assert {[string match $pattern $result]}
}
}
}
[21/66 done]: integration/convert-zipmap-hash-on-load (1 seconds)
Testing integration/logging
[err]: Crash report generated on SIGABRT in tests/integration/logging.tcl
Expected [string match *STACK TRACE* ### Starting server for test
24174:C 08 May 2022 16:25:35.302 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
24174:C 08 May 2022 16:25:35.302 # Redis version=6.2.7, bits=64, commit=00000000, modified=0, pid=24174, just started
24174:C 08 May 2022 16:25:35.302 # Configuration loaded
24174:M 08 May 2022 16:25:35.303 * monotonic clock: POSIX clock_gettime
24174:M 08 May 2022 16:25:35.304 # A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.2.7 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 23118
| `-._ `._ / _.-' | PID: 24174
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
24174:M 08 May 2022 16:25:35.304 # Server initialized
24174:M 08 May 2022 16:25:35.306 * Ready to accept connections
24174:M 08 May 2022 16:25:35.306 * The server is now ready to accept connections at /home/sas/source/redis-6.2.7/tests/tmp/server.19128.18/socket
24174:M 08 May 2022 16:25:35.415 - Accepted 127.0.0.1:51920
24174:M 08 May 2022 16:25:35.415 - Client closed connection
24174:M 08 May 2022 16:25:35.424 - Accepted 127.0.0.1:51922
### Starting test Crash report generated on SIGABRT in tests/integration/logging.tcl
=== REDIS BUG REPORT START: Cut & paste starting from here ===
24174:M 08 May 2022 16:25:35.429 # Redis 6.2.7 crashed by signal: 6, si_code: 0
24174:M 08 May 2022 16:25:35.429 # Killed by PID: 24264, UID: 1000
24174:M 08 May 2022 16:25:35.429 # Crashed running the instruction at: 0x2b0dfcfe6483] (context: type eval line 7 cmd {assert {[string match $pattern $result]}} proc ::test)
The following logs are from the stdout file (redis-6.2.7/tests/tmp/server1.log.19128.17/stdout). The STACK TRACE is available.
=== REDIS BUG REPORT START: Cut & paste starting from here ===
24174:M 08 May 2022 16:25:35.429 # Redis 6.2.7 crashed by signal: 6, si_code: 0
24174:M 08 May 2022 16:25:35.429 # Killed by PID: 24264, UID: 1000
24174:M 08 May 2022 16:25:35.429 # Crashed running the instruction at: 0x2b0dfcfe6483
24174:signal-handler (1652007335) Received SIGTERM scheduling shutdown...
------ STACK TRACE ------
EIP:
/lib64/libc.so.6(epoll_wait+0x33)[0x2b0dfcfe6483]
Backtrace:
/lib64/libpthread.so.0(+0xf5d0)[0x2b0dfccdb5d0]
/lib64/libc.so.6(epoll_wait+0x33)[0x2b0dfcfe6483]
src/redis-server 127.0.0.1:23118(aeProcessEvents+0x132)[0x47d802]
src/redis-server 127.0.0.1:23118(aeMain+0x1d)[0x47dc1d]
src/redis-server 127.0.0.1:23118(main+0x503)[0x47a673]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x2b0dfcf0a3d5]
src/redis-server 127.0.0.1:23118[0x47a93a]
- Getting the following warning message on the redis startup log.
A key '__redis__compare_helper' was added to Lua globals which is not on the globals allow list nor listed on the deny list.
Please let me know if anything needs to worry about on this? or Can safely ignore it?
Additional information
- CentOS 7.6
- GCC 4.9.2
- OpenSSL 1.1.1k
Comment From: enjoy-binbin
or Can safely ignore it?
thanks for the report, yes you can ignore it right now (the STACK TRACE is available) It looks like this test is not stable, i will take a deep look later
As for __redis__compare_helper, I didn't find a reference to it, seem harmless
Comment From: ganeshkumarganesan
@enjoy-binbin Thank you.
Comment From: enjoy-binbin
ohh, the STACK TRACE timing issue was actually fix in #9910, unstable branch
Comment From: oranagra
Thanks @enjoy-binbin.
As you indicated the sporadic test failure is already fixed in the later version.
The warning about __redis__compare_helper is something we overlooked, but we now conclude it is harmless. @MeirShpilraien will post more details about it later in a PR that fixes the warning.
Meanwhile, we can close this one...
Comment From: ganeshkumarganesan
As for __redis__compare_helper, I didn't find a reference to it, seem harmless
https://github.com/redis/redis/blob/e6f67092f8d4d81761a60c46011d1ff1dc3c2628/src/scripting.c#L434-L435
@enjoy-binbin The __redis__compare_helper reference is available in the 6.2.7 branch. This has been removed in this commit. Hence not available in the unstable branch.
On the server startup or script flush, we are getting the warning message in redis logs. Can you please check and confirm whether it is harmless?
https://github.com/redis/redis/blob/e6f67092f8d4d81761a60c46011d1ff1dc3c2628/src/scripting.c#L1269-L1271
Comment From: enjoy-binbin
ok, i see now. i search it in unstable branch, so we should add it in redis_api_allow_list or something like that
Comment From: oranagra
@ganeshkumarganesan it seems harmless. Meir will post more details later..
in short, it was added in https://github.com/redis/redis/commit/2c861050c1 in order for scripts to sort the output of SORT.
and should have later been removed in https://github.com/redis/redis/commit/36741b2c818a95e8ef167818271614ee6b1bc414 (when the S i.e. to-sort flag was removed from SORT)
Comment From: oranagra
https://github.com/redis/redis/pull/10701