We were bitten by a nasty bug due to redis timing out on our client. We ultimately found out that the culprit was ntpdate moving system time forward of some days, so that all timeouts started elapsing much earlier than they were needed. I know we should have restarted all sensitive services on such a time jump, but luckily there are new system calls which try to avoid this kind of problems. Could we use e.g. clock_gettime() instead of time() to keep a monotonic server time to be used in all elapsed time calculations?
I'd be willing to propose a patch if you think it would be interesting.
TIA
Comment From: antirez
I'm not sure I can do this, because this system call is not generally available AFAIK.
But what I can easily do is, at least, to make Redis aware that a strange time shift happened and log the event. Btw there are many reasons why I can't do the monotonically clock stuff. For instance semantically in Redis an EXPIRE set the time at which a key will expire, because if you save the DB, it will still expire at that date.
Is not really "expire after 10 seconds" even if this is what actually happens. It's instead: - EXPIRE -> Set the date at which a key will expire. - Expiring process -> Check if key reached that date, remove it if needed.
And indeed, both RDB, AOF and Replication are handled in this way, converting expires into EXPIREAT commands.
Comment From: beatgammit
I just ran into a similar problem with snapshotting: if the system time goes backwards, snapshotting is still relative to the system time when redis started (or more likely the last snapshot).
I admit this is a pretty fringe case, but I've been bitten by this on an embedded device that can lose power or have the system time change at any time.
Comment From: amv
@antirez I believe you mentioned somewhere along the Kleppmann's redlock analysis discussion that you were reconsidering the monotonic clock stuff.
Has the general availability of the monotonic clock api changed in the past 6 years so that this would be more feasible?
How have your thoughts on the other things mentioned here changed since then? Is there anything one can do to expedite this change so that we could trust redlock locks better in the face of leap seconds, accidental ntpdate runs or other operational mysteries that might affect the realtime clock?
Comment From: amv
I believe that for the redlock use case (without persistence) it would be enough to just provide a configuration parameter that would switch between the system using realtime and monotonic time, with no alterations to the replication logic needed.
If the system time would still remain as the default option, nothing should break in existing installations. And this being an opt-in feature, one could just not allow turning it on when the underlying system does not provide a monotonic clock api.
Could this be at least a first step to take towards supporting more safety when using redis as a building block for distributed systems? If there exists more interest in this, people could propose additional "clock modes" to be added in the future.
Examples might include opting for CLOCK_TAI or CLOCK_MONOTONIC_RAW, and also some more complicated schemes which do monotonic vs realtime comparisons when importing and exporting EXPIREAT. But while these are being formulated, having the possibility to switch to monotonic time would be of great help :)
Comment From: amv
It seems that Linux provides a lot of (Linux only) clock options nowadays, and just being able to pick the one you want to use in your system could be a bug step further in the distributed systems building block space:
https://www.systutorials.com/docs/linux/man/2-clock_gettime/
Especially CLOCK_BOOTTIME looks very promising for what it claims to do :)
Comment From: sj82516
I am curious what is this issue current state when I saw the discussion about Redlock.
Comment From: antirez
Hello, trying to resurrect this thread. I think that we could do that even if there is just a Linux-only interface to start, but with proper #ifdef-ing of the function, so that we provide gettimeofday() if we have no other way. However this requires quite some work: I think it's worth it, but is not just a quick PR that will substitute one call with the other, we need to understand exactly how every call fetching the time is used in Redis right now, and substitute it with the monotonic version if possible. Potentially we also need to have two versions of the cached time (server.mstime & co), one normal and one monotonic.
Comment From: yossigo
Closing as this was addressed by #7644 (and work to migrate everything to it is in progress).