There are two related metrics I see tracking_total_keys and tracking_total_items.
The first one (tracking_total_keys) can be hard limited by tracking-table-max-keys config setting. However, I cannot find any documentation about second metric tracking_total_items. As I can guess, this is total amount of invalidation table entries which is multiplication of total tracking keys and connected clients, am I correct? If so, does these tracking items are dropped when some client disconnects?
Also, there is no any configurable limit for tracking_total_items. In our application, we see tracking_total_keys near 6M keys, while tracking items reached 150M (we have 130 connected clients). As I guess, there can be implicit limit for tracking_total_items depending on tracking clients: tracking_total_keys * tracking_clients. However, such dynamic limiting looks not reliable. For example, what happen if there were 1000 tracking clients connected?
Is it possible to cause memory outage on Redis server, if too many clients connect with client tracking enabled and register keys in invalidation table?
And also, when have tracking items removed form invalidation table (except normal way, when write command issued and fires invalidation)? I guess, this should happen when tracking client disconnects. But does this happen immediately when client disconnects. Or it happens lazily in some way?
Thanks, Dima.
Comment From: huangzhw
- If a client tracking a key,
tracking_total_itemswill +1. Tracking items are dropped in a lazy way. So these tracking items will not drop. - Yes.
- Yes. It will.
- Invalidation happens lazily. So when write command issued, these items are deleted. Or when reached
tracking-table-max-keys, server starts to invalidate keys.
Comment From: rd162
@huangzhw thanks for you answers. I have yet one question: what happens when Redis reached its memory limit or exhausted all available host memory? Will Redis try to force remove "stale tracking items" to free memory?
Comment From: madolson
@rd162 It will not, although I think that would be a good idea :) Maybe we should have some concept of a "tracking-table-soft-key-limit" so that if we've exhausted memory, we will try to remove items from the tracking table.
Comment From: yang-dev7
@huangzhw @madolson Is there any solution to this problem now? We are currently conducting a POC verification of CSC and have also discovered this problem. After enabling tracking, the memory usage of the redis server machine has been increasing, as shown in the following figure
