We are experiencing really weired behavior of Redis server, which is randomly appearing (at least it looks like that). When problem happens, Redis is locked, not responding and even won't stop (must be killed with signal 9). This is appearing in logs until server is killed and started again:
20704:M 06 Nov 14:41:17.090 * 10 changes in 300 seconds. Saving... 20704:M 06 Nov 14:41:17.091 * Background saving started by pid 5333 5333:C 06 Nov 14:41:17.091 # Failed opening the RDB file root (in server root dir /var/spool/cron) for saving: Read-only file system 20704:M 06 Nov 14:41:17.191 # Background saving error
Note that error 'Read-only file system' means that systemd is not allowing Redis to write to that directory (it is running in different mount namespace, the filesystem is really mounted in read-write mode).
I was also strace-ing it and Redis is really trying to create temporary files in /var/spool/cron and is ignoring settings in configuration file: dbfilename dump.rdb dir /var/lib/redis
System is Debian Jessie, package is from dotdeb.org, 64bit OpenStack VPS.
Any hints?
Comment From: itamarhaber
Is your Redis server accessible to the public? If so, this could be an attempted attack.
On Nov 6, 2016 5:11 PM, "azurit" notifications@github.com wrote:
We are experiencing really weired behavior of Redis server, which is randomly appearing (at least it looks like that). When problem happens, Redis is locked, not responding and even won't stop (must be killed with signal 9). This is appearing in logs until server is killed and started again:
20704:M 06 Nov 14:41:17.090 * 10 changes in 300 seconds. Saving... 20704:M 06 Nov 14:41:17.091 * Background saving started by pid 5333 5333:C 06 Nov 14:41:17.091 # Failed opening the RDB file root (in server root dir /var/spool/cron) for saving: Read-only file system 20704:M 06 Nov 14:41:17.191 # Background saving error
Note that error 'Read-only file system' means that systemd is not allowing Redis to write to that directory (it is running in different mount namespace, the filesystem is really mounted in read-write mode).
I was also strace-ing it and Redis is really trying to create temporary files in /var/spool/cron and is ignoring settings in configuration file: dbfilename dump.rdb dir /var/lib/redis
System is Debian Jessie, package is from dotdeb.org, 64bit OpenStack VPS.
Any hints?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/antirez/redis/issues/3594, or mute the thread https://github.com/notifications/unsubscribe-auth/AFx1_KhYzEZupnh0VfATUQqnHbBfH3YEks5q7e4PgaJpZM4KqlBk .
Comment From: badboy
The configuration can be changed using CONFIG SET, which is likely what happened.
Given the directory it's even more likely some malicious intruder did this. Make sure your server is not accessible from the internet and/or add password-protection.
Consider your server compromised and replace it immediately.
Comment From: azurit
So anyone can change configuration online without touching config file?
Comment From: badboy
If you open it up to the whole wide internet, everyone can access everything stored in Redis, yes.
Comment From: azurit
One thing is access the data and another is changing the configuration of the server. I think this should be disabled by default. Anyway, what's the reason to change things like database filename and path online? This shouldn't be allowed at all.
Comment From: badboy
Then deactivate the config command.
Comment From: azurit
badboy what's the point in changing database file name and path with config command? Has anyone done this with a sensible reason? This change even isn't persistant, what's the point in setting database location valid until next server restart?
Comment From: badboy
Not everyone runs Redis with a configuration file in the first place. Dynamically reconfiguring has its use cases.
You can persist the dynamically changed configuration to disk using CONFIG REWRITE
Comment From: itamarhaber
@azurit defaults are meant to be changed by responsible adults to their proper values on a per-deployment case - no one default value fits all cases, so Redis' are designed to work out of the box but are by no means relevant to you. Take the time and read through the thoroughly-documented redis.conf file and customize it to suit your needs.
In any case, protected-mode is a new default (in v3.2+) that is designed to prevent this types of attacks due to misconfiguration. I recommend, if possible, that you upgrade to it.
Comment From: azurit
I'm using 3.2.5 and protected-mode is and was enabled. I was binding to a specified IP addresses so it, probably, wasn't triggered. Or it don't work at all (or something completely different happend but i bet the problem was caused by changed configuration by someone from the net).
Comment From: azurit
and, by the way, defaults should be meant to be safe for all and not allowing things like creating new files 'anywhere' in the system by simply opening a port.
Comment From: SrujithPoondla
Could someone explain how to use protected mode in the redis cluster. If i set the protected mode to yes then the remaining nodes in the cluster were not able to find each other.
Comment From: cuongn265
@azurit have you resolved this? I have the same problem here, my redis dir and dbfilename is change to
redis-cli config get dir
1) "dir"
2) "/var/spool/cron"
redis-cli config get dbfilename
1) "dbfilename"
2) "root"
I'm using multiple AWS EC2 instances and connect to 1 redis server, and it occurred again and again when I restart sidekiq server.
May it's related with #4791
Comment From: azurit
@cuongnm265 yes i was able to resolve it, it appeard that some kind of bots were trying to somehow hack the server using Redis by online changing it's configuration (with CONFIG command). I did: bind 127.0.0.1 protected-mode yes rename-command CONFIG ""
As Redis doesn't have any priviledge separation, anyone can run command CONFIG and, at least, doing a DoS attack to Redis. So i decided to completely disable CONFIG command, even for local users.
Comment From: steevhise
This has been happening to us too, and i find it hard to believe it's because of bots. are there really bots swarming around trying port 6379 of random servers? and if they wanted to cause havoc wouldnt they do more than just change the write directory? for instance with that kind of power they could delete everything in the store. or change the data in order to cause chaos and confusion. why just change the directory?
So, does anybody have an alternate explanation? or evidence that it's really malicious?
Comment From: misiek08
Those are scanners and they try many ports on whole IP subnets.
They try to inject cron settings or run some bash script to install software and make money with yours machine CPU power.
It's funny, because some of the script in wild will even block traffic on firewall, so they install malicious soft and secure your server from following attacks.
sob., 21.07.2018, 02:59 użytkownik Steev Hise notifications@github.com napisał:
This has been happening to us too, and i find it hard to believe it's because of bots. are there really bots swarming around trying port 6379 of random servers? and if they wanted to cause havoc wouldnt they do more than just change the write directory? for instance with that kind of power they could delete everything in the store. or change the data in order to cause chaos and confusion. why just change the directory?
So, does anybody have an alternate explanation? or evidence that it's really malicious?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/antirez/redis/issues/3594#issuecomment-406759109, or mute the thread https://github.com/notifications/unsubscribe-auth/AAbp9l0RjAuPGu605q8mhpM8-y601i4Bks5uIn0OgaJpZM4KqlBk .
Comment From: azurit
In our case, the bots were changing the write directory and were trying to inject a cron script but i don't understand how this could work - is there anyone running redis under the cron or even root user?
Comment From: misiek08
Most people are using Redis on user, where cron will work. They don't need root - scripts often download binary file and run it. It doesn't need open ports or privileges bigger than Redis user.
Comment From: azurit
misiek08 you are wrong, common users don't have write access to directory, where crons are stored (usually /var/spool/cron/crontabs). Crons can be set up only by using a crontab command, which is 'suid root' or 'sgid crontab'.
Comment From: misiek08
@azurit maybe attackers aim only into people, who run Redis on root. Every user has access to his file in /var/spool/cron/crontabs so maybe the path is incorrect (maybe some script kiddies).
Comment From: ghost
Same thing happened, just 7 minutes after launching a server )))):
Script downloads crypto miner, indeed secures the port through iptables :))))).
iptables -I INPUT -p TCP --dport 6379 -j REJECT
iptables -I INPUT -s 127.0.0.1 -p tcp --dport 6379 -j ACCEPT
iptables-save
Happened on a docker instance, as I failed to secure docker ports in ubuntu while was in a rush to create a demo server with dummy data, so Redis was opened and only after 7 minutes it got sacked.
This is a script itself https://pastebin.com/tRxfvbYN (I reported abuse already)
For anybody who will run into the same issue with docker.
Don't be fooled, docker ignores ufw unless configured properly. Steps to force this bloke to obey rules:
- create the file /etc/docker/daemon.json with contents
{
"iptables": false
}
- vim /etc/default/docker (or where you put that)
Uncomment DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 —iptables=false"
P.S. add --iptables=false
- Edit before-rules: /etc/ufw/before.rules
#Add before *filter
*nat
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING ! -o docker0 -s 172.17.0.0/16 -j MASQUERADE
COMMIT
- Restart docker daemon or even better reboot.
- Ensure that ufw is enabled and allows only ports you want to be exposed. e.g. 443
- don't be shy to use nmap on your server to verify that all ports are closed:
nmap -Pn Your_server_ip_address
Comment From: redgenesisadmin
this can be solved by activating UFW firewall, allow only those machines who are autherised to access.
Comment From: lyhistory
I've tested injection through the ‘loophole', my conclusion is that as long as you don't open your port to the external network and run the redis using root user at the same time, you will lower the risk, but not 100% guaranteed, they can make use of other folders like the web host folder as long as the redis user has permission accessing it, so the issue is about your whole network architecture, you need to put these kind of internal services in a safe zone.
Comment From: ArtyomKozyrev8
I've faced the similar issue today. Can somebody give any suggest?
Comment From: myysophia
redis 3.2.11
This has been happening to us too。
19103:M 13 Apr 09:05:01.016 * Background append only file rewriting started by pid 39423
19103:M 13 Apr 09:09:36.445 * AOF rewrite child asks to stop sending diffs.
39423:C 13 Apr 09:09:36.454 * Parent agreed to stop sending diffs. Finalizing AOF...
39423:C 13 Apr 09:09:36.454 * Concatenating 3.23 MB of AOF diff received from parent.
39423:C 13 Apr 09:09:36.459 * SYNC append only file rewrite performed
39423:C 13 Apr 09:09:36.797 * AOF rewrite: 2648 MB of memory used by copy-on-write
19103:M 13 Apr 09:09:37.495 * Background AOF rewrite terminated with success
19103:M 13 Apr 09:09:37.495 * Residual parent diff successfully flushed to the rewritten AOF (0.02 MB)
19103:M 13 Apr 09:09:37.496 * Background AOF rewrite finished successfully
19103:M 20 Apr 15:50:01.060 * Background append only file rewriting started by pid 23589
23589:C 20 Apr 15:53:14.983 # Write error writing append only file on disk: No space left on device
19103:M 20 Apr 15:53:15.439 # Short write while writing to the AOF file: (nwritten=4, expected=118)
19103:M 20 Apr 15:53:15.476 # AOF write error looks solved, Redis can write again.
19103:M 20 Apr 15:53:16.309 # Background AOF rewrite terminated with error
19103:M 20 Apr 15:55:00.743 * Background append only file rewriting started by pid 24235
24235:C 20 Apr 15:55:58.605 # Write error writing append only file on disk: No space left on device
19103:M 20 Apr 15:55:58.826 # Short write while writing to the AOF file: (nwritten=17, expected=69)
19103:M 20 Apr 15:55:59.334 # AOF write error looks solved, Redis can write again.
19103:M 20 Apr 15:55:59.930 # Background AOF rewrite terminated with error
19103:M 20 Apr 16:00:00.731 * Background append only file rewriting started by pid 24983
24983:C 20 Apr 16:00:58.088 # Write error writing append only file on disk: No space left on device
19103:M 20 Apr 16:00:59.240 # Background AOF rewrite terminated with error
i have a redis cluster, only the 6379 had attacked. it‘s easy to attacked when the port scanned
The intruder may just be a warning，because they can do ‘redis-cli -p 63** FLUSHDB’ without password .a matter for rejoicing;
Comment From: ArtyomKozyrev8
redis 3.2.11 This has been happening to us too。
19103:M 13 Apr 09:05:01.016 * Background append only file rewriting started by pid 39423 19103:M 13 Apr 09:09:36.445 * AOF rewrite child asks to stop sending diffs. 39423:C 13 Apr 09:09:36.454 * Parent agreed to stop sending diffs. Finalizing AOF... 39423:C 13 Apr 09:09:36.454 * Concatenating 3.23 MB of AOF diff received from parent. 39423:C 13 Apr 09:09:36.459 * SYNC append only file rewrite performed 39423:C 13 Apr 09:09:36.797 * AOF rewrite: 2648 MB of memory used by copy-on-write 19103:M 13 Apr 09:09:37.495 * Background AOF rewrite terminated with success 19103:M 13 Apr 09:09:37.495 * Residual parent diff successfully flushed to the rewritten AOF (0.02 MB) 19103:M 13 Apr 09:09:37.496 * Background AOF rewrite finished successfully 19103:M 20 Apr 15:50:01.060 * Background append only file rewriting started by pid 23589 23589:C 20 Apr 15:53:14.983 # Write error writing append only file on disk: No space left on device 19103:M 20 Apr 15:53:15.439 # Short write while writing to the AOF file: (nwritten=4, expected=118) 19103:M 20 Apr 15:53:15.476 # AOF write error looks solved, Redis can write again. 19103:M 20 Apr 15:53:16.309 # Background AOF rewrite terminated with error 19103:M 20 Apr 15:55:00.743 * Background append only file rewriting started by pid 24235 24235:C 20 Apr 15:55:58.605 # Write error writing append only file on disk: No space left on device 19103:M 20 Apr 15:55:58.826 # Short write while writing to the AOF file: (nwritten=17, expected=69) 19103:M 20 Apr 15:55:59.334 # AOF write error looks solved, Redis can write again. 19103:M 20 Apr 15:55:59.930 # Background AOF rewrite terminated with error 19103:M 20 Apr 16:00:00.731 * Background append only file rewriting started by pid 24983 24983:C 20 Apr 16:00:58.088 # Write error writing append only file on disk: No space left on device 19103:M 20 Apr 16:00:59.240 # Background AOF rewrite terminated with errori have a redis cluster, only the 6379 had attacked. it‘s easy to attacked when the port scannedThe intruder may just be a warning，because they can do ‘redis-cli -p 63** FLUSHDB’ without password .a matter for rejoicing;
@myysophia hello check this answer on Stackoverflow, it will solve the issue: link