We have a Redis setup with a set of 3 redis instances and a set of 3 sentinels.

While we've not run into any issues with upgrading in the past, there were some questions about forward- and backward-compatibility guarantees during the upgrade process. The redis docs and the release notes are rather light on guidance here (at least we didn't find any details).

As we're now approaching the redis 5 to 6 upgrade, we wanted to make sure we know which directions we can roll back, in case there are any issues.

  • What order should processes be upgraded in? Sentinels first? Or redis instances first?
  • After upgrading sentinels, is it possible to safely downgrade again?
  • Is it possible to run sentinels with mixed versions (e.g. some 5, some 6)?
  • What about downgrading redis? Are there compatibility concerns with the file format, replication protocol?
  • Is it possible to run redis with mixed versions?

We've started running some tests for several of these scenarios. It might be useful for others to have an explicit statement for which upgrade paths are supported, including rollbacks.

Comment From: yossigo

@igorwwwwwwwwwwwwwwwwwwww Great topic, unfortunately I have to agree with you that documentation is lighter than it should be around this. It's an on-going and endless effort.

You can run mixed Redis versions, as long as you have a Vold master and Vnew replica (replica is newer and backwards compatible with the master). It's not a recommended permanent state but you anyway have to go through it to do an online upgrade with no downtime.

Downgrading is NOT a good idea, and in many cases not even possible: * RDB formats may change, so an old version will refuse to load a newer RDB or fail to full-sync * Commands change so AOF files may be incompatible and leave you in an undefined state

Sentinel didn't see significant changes AFAIR between 5.x and 6.x, so I think as long as you don't use any new features (like ACL support, recent hostnames support, etc.) you should probably be safe in any order you choose, but I urge to properly test this in advance.

Comment From: igorwwwwwwwwwwwwwwwwwwww

For the record, I will link our downstream issue, which includes a reference to our own upgrade testing jig: redis-upgrade-harness.

Our testing so far suggests that a downgrade from 6.0.10 to 5.0.10 is possible. But there may be issues our testing hasn't caught yet.

Our upgrade plan so far is as follows:

  • A rolling upgrade of all sentinels.
  • Upgrade a single redis node from 5 to 6.
  • Promote that node to master.
  • Run in this state for a few hours, with two replicas still running the old version.
  • If anything goes wrong, we have the option to promote one of the older nodes (effectively rolling back the upgrade).
  • Once we're confident that things are working, we can upgrade the two replicas as well.

Comment From: yossigo

@igorwwwwwwwwwwwwwwwwwwww This doesn't break completely because Redis 5 and Redis 6 have the same RDB version. You couldn't have pulled this off with Redis 4 and Redis 5, for example, and that may be the case in the future as well.

Having a newer master and an older replica means commands may be mutating the master and replica datasets in different ways, making them inconsistent.

The most obvious case would be commands that simply do not exist on the replica and fail. But there could be other, more subtle differences. Whenever we modify a command behavior we're doing our best to make it backwards compatible for exactly this reason, but we always assume replica >= master.

Bottom line, I advise against that.

If you want to get some mileage from the new version, you can run it as a replica alongside the older version replicas and master. Yes, you don't get full coverage this way but you also don't risk an inconsistency.