Allow to set an expiration on hash fields. Already discussed at #1042 and #167

Comment From: zemochen

It's 2020-05-18, has this feature implementation?

Comment From: zuiderkwast

This idea was rejected by antirez back in #167, #242 and #1042. Later, #3192, #8643, #8269, #9004 and this one have been opened. Now, there is a different leadership and it's on the agenda again. It seems that it can be accepted if it can be done in an efficient way. Some of those issues contain some interesting comments explaining the implementation problems.

There is a module which can do this: https://github.com/alibaba/TairHash. Someone needs to evaluate how this module works and if this is the way we want it to work in native Redis too.

  • [ ] Evaluate TairHash
  • [ ] Discuss alternative solutions and agree on how to implement it in Redis
  • [ ] Implement

Let's use this issue to discuss this further.

Comment From: ThaDaVos

Any progress?

Comment From: jilinskycloud

any progress on expiring hash in keys?

Comment From: chenyang8094

Mention something I can think of.

I think there are the following difficulties in implementing hash with expiration in redis:

  • Existing commands are difficult to extend the expire parameter, such as HSET key field value EX secondsorHINCRBY key field increment EX seconds cannot be realized. If new commands such as HEXPIRE field seconds are added, the user will have to use the pipeline method to combine HSET and HEXPIRE to reduce network overhead.
  • In order to support this feature, the expire information must be stored inside the hash structure, which will cause the RDB encoding format inside to change, and it will no longer be forward compatible.
  • We also need to consider the memory overhead brought by the introduction of expire, especially when our active expiration is not efficient
  • The hash structure supports OBJ_ENCODING_LISTPACK and OBJ_ENCODING_HT two encoding formats in the memory. In order to support expire, they need to be big modified or compromised.
  • Active expiration efficiency is also a problem. At present, redis uses scan to implement the active expiration of the main key. If the hash still does not support the field sorting according to expiration, it is conceivable that the field-level active expiration efficiency will be worse, but the advantage is that it is relatively simple to implement.
  • If we want to improve the expiration efficiency, a sorting structure such as min heap/ziplist/sort tree, etc. is necessary, but this will lead to the complexity of index maintenance. At the same time, in order to reduce the memory overhead caused by the field copy, only the pointer of the field should be used in the index, which will cause the hash to not use the OBJ_ENCODING_LISTPACK compression encoding format.
  • There may be other problems I didn't expect. But I think it might be a better choice to support a completely new data type.

Comment From: tzographos

Is the hash field expiration still an issue ? We are 10 YEARS since the original request here in 2011. There have been countless questions throughout the decade from people asking for this functionality and still nothing.

I think it's time we switched to the KeyDB fork which offers this and much more with higher performance.

Comment From: itamarhaber

Thanks for summing it all up @chenyang8094 - truly, this is quite a challenge to get right. Mainly the points involving "effectiveness" and "active".

As an observation, it should be quite easy to implement a non-complex lazy expiry in Hashes (and probably the rest of the nested types, excluding Streams). The thing is, it won't really help when there's memory pressure.

P.S. ever since I saw my first Star Trek episode (so ... 40+ years?) I wanted Scotty to beam me up.

Comment From: kurogai

Hi, any updates?

Comment From: tillkruss

I insist on this too (jk). Even with some extra overhead in Redis, we'd use expiring hashes a lot!

Comment From: romange

This feature has been implemented in Dragonfly: https://github.com/dragonflydb/dragonfly/pull/817

Comment From: moticless

Heads up, we've started implementing hash field expiration, and it's planned for an upcoming release.

Comment From: deefdragon

Hash field expiration has been merged in as of Redis 7.4 and this and the other tickets can probably be closed.

Before they are tho, I'm wondering, would it be possible to use the techniques used to support hash field expiration on sets & sorted sets as well? I assume there is a lot of additional complexity with at least zsets given they are sorted, but one can hope.