Redis [BUG] if a test that sets maxmemory fails, it doesn't unset maxmemory limits

Describe the bug

if a test that sets maxmemory (say to 1), and then unsets to 0 at the end, fails, it doesn't seem to reset the value itself.

this is hidden when the suite brings up and down redis instances, but when run against an external instance is visible.

i'm seeing it due to a problem I'm having running "Consistent eval error reporting" in scripting.tcl (not a redis bug)

To reproduce

run against an external instance and force a test that sets maxmemory to 1 to fail and then run another test that doesn't expect any real memory limits that should pass

Expected behavior

said second test will now fail

Additional information

Any additional information that is relevant to the problem.

Comment From: oranagra

sadly, many of our tests depend on each other, so when one of them fails (or is skipped), the others that follow can't be trusted. this is specifically problematic with external servers, that not only shared between different test units, but also in repeated test suite runs. that said, i'm not sure we wanna make the effort in fixing it. we can not even write a mechanism to detect config changes and revert them when a test fails. we can maybe do something that resets the server between units in external mode (we actually have some of it there already), but it might be problematic to make it air tight considering external servers don't always support all configs and tests.

Comment From: sjpotter

one thing that would be interesting re this is

a) making the ability to setup and tear down "external" servers something that can be plugged into the suite.

i.e. whenever start_server is called, it does whatever the "plugin" it requires of it (and perhaps even provides the port the to connect on).

b) at least for maxmemory, it could be interesting if the framework was able to understand the tag of "needs:config-maxmemory" to mean that maxmemory should be set to 0 at the end of the test (which would remove doing that in every single test and also help find places where one should be using that tag).

with that said, perhaps the gain isn't worth it (/ perhaps a pain to do in tcl).

Comment From: oranagra

it would probably be easy to reset maxmemory and others in start_server on external mode. but it won't solve your problem with a failing test messing up other tests that follow. and adding an implicit reset of maxmemory between tests would break some tests. i don't think any of that worth it.