The problem/use-case that the feature addresses Running Redis 7.0.9 in a container. The novel logging format Redis currently employs presents friction to slurping its log output into monitoring.

Description of the feature A configuration option that allows us to configure Redis to output its logs in at least one of JSON and logfmt. Either of these formats are eminently grokkable by telemetry collection systems like fluentd, telegraf, promtail. Having this option would make Redis more fluidly friendly with our systems, and allow us to more easily and reliably alert on errors and warnings.

Thanks!

Comment From: zuiderkwast

Everywhere throughout the Redis source code, logging is done with printf-like functions. It will be a very large change to change all those. I don't think it is acceptable to do it.

In a central place in the code is the formatting which adds the timestamp and a few other thing to each line. Here it is easy to add some conditional code to output it differently.

Here is an example of some log lines:

447467:M 03 Jan 2024 16:20:50.708 * Server initialized
447467:M 03 Jan 2024 16:20:50.708 * Ready to accept connections tcp
447467:M 03 Jan 2024 16:20:50.708 * Ready to accept connections unix
447467:M 03 Jan 2024 16:20:50.719 - Accepted 127.0.0.1:40327

First is the process id. Then is the role (M = master, S = replica, X = sentinel, C = RDB / AOF writing child). Then timestamp. Then a symbol (# = warning, * = notice, - = info, . = debug). After that is a log entry generated using print-like code like serverLog(LL_VERBOSE,"Accepted %s:%d", cip, cport); all over the source code.

What would be relatively each to achieve is change the formatting for the fixed fields, but keep the printf-like messages as they are, i.e. some formatting like this would be easy to achieve with a config:

pid=447467 role=M time=2024-01-03T16:20:50.719 level=info message="Accepted 127.0.0.1:40327"

Would that be good enough?

Comment From: sundb

Another alternative is to using like filebeat or logstash to format the logs and feed them to the monitor system.

Comment From: enspritz

@zuiderkwast I understand the refactoring load, and I don't presently have the latitude to take it on myself. That last formatting example resemblant of logfmt you shared, at first glance it would indeed be good enough.

I'm no authority on logfmt, but as long as the telemetry collectors that grok logfmt will accept this style of output, then I imagine the process of integrating Redis into log collection/monitoring/alerting will merely be a matter of instructing the collector to parse loglines as logfmt and away we go!

Comment From: fede843

I would really like to have this feature too.

Comment From: zuiderkwast

Feel free to implement it and open a PR. Then we can also see the complexity of it and take a decision. Personally I like the logfmt idea. We don't have JSON in Redis, so we can avoid adding the dependency. We probably need a simple function to do string escaping for the log message though.

Comment From: azuredream

Hi, I opened a PR to print log in logfmt style.

pid=12073 role_char=M timestamp="10 Jan 2024 23:29:03.839" level= message="Server initialized" pid=12073 role_char=M timestamp="10 Jan 2024 23:29:03.839" level= message="Ready to accept connections tcp" ^C12073:signal-handler (1704947359) Received SIGINT scheduling shutdown... pid=12073 role_char=M timestamp="10 Jan 2024 23:29:19.433" level= message="User requested shutdown..." pid=12073 role_char=M timestamp="10 Jan 2024 23:29:19.433" level= message="Saving the final RDB snapshot before exiting." pid=12073 role_char=M timestamp="10 Jan 2024 23:29:19.436" level=* message="DB saved on disk" pid=12073 role_char=M timestamp="10 Jan 2024 23:29:19.436" level=# message="Redis is now ready to exit, bye bye..."

I'll run more test tomorrow. I'm newcomer to this repo. I'd appreciate it if you could give me some advice on this. Thanks!

Comment From: azuredream

Please review the sample output to see if that meets your requirements. A question is that do you prefer role_char(M) or full role name (master). Same question for verbose level. https://github.com/redis/redis/pull/12934#issuecomment-1890072144

Comment From: palmarisiva

Is there any plan to include JSON format in the configuration option? As per https://github.com/redis/redis/pull/12934, configuration changes using logfmt code changes are committed.

Comment From: zuiderkwast

@palmarisiva There is no plan, at least not yet. It seemed to me that either logfmt or JSON would be enough. I suggested logfmt because it seemed simple, but I guess JSON would be very similar to implement. Why you need JSON specifically?

Comment From: palmarisiva

@palmarisiva There is no plan, at least not yet. It seemed to me that either logfmt or JSON would be enough. I suggested logfmt because it seemed simple, but I guess JSON would be very similar to implement. Why you need JSON specifically?

@zuiderkwast Our application uses logstash to process logs which supports JSON format. JSON format is used as a common logging mechanism by all the applications. Hence, expecting the JSON format in Redis too.

Comment From: ksemele

@zuiderkwast we have a lot of apps in our production, and all of them can use JSON as the log format. We use fluent bit for all our apps and third-party apps (such as Kafka, RabbitMQ, etc.). Why do we need to write custom parsers or use a different log collector only for Redis? I think it's a great point for a roadmap.

Comment From: troywweber7

In case this helps anyone looking for a local-run solution... if you have node.js installed. I created a file named redis-json.mjs with the following content.

#!/usr/bin/env node

import { createInterface } from 'node:readline'
import { stdin as input, stdout as output } from 'node:process'

const logs = createInterface({ input })
const logPattern = /^(?<pid>\d+):(?<role>[A-Z]{1}) (?<date>\d+ \w+ \d+ \d{2}:\d{2}:\d{2}\.\d{3}) (?<level>[#*-.]{1}) (?<message>.+)$/
const roles = {
    M: 'master',
    S: 'replica',
    X: 'sentinel',
    C: 'RDB / AOF writing child'
}
const levels = {
    '#': 'warning',
    '*': 'notice',
    '-': 'info',
    '.': 'debug'
}

logs.on('line', (line) => {
    const match = logPattern.exec(line);
    if (!match) return;

    const pid = parseInt(match.groups.pid);
    const role = roles[match.groups.role];
    const date = new Date(match.groups.date);
    const level = levels[match.groups.level];
    const message = match.groups.message;

    const jsonLine = JSON.stringify({ pid, role, date, level, message })
    output.write(jsonLine + '\n')
});

Then I just pipe the logs with <redis-command-producing-logs> | ./redis-json.mjs. The output should look something like the following.

{"pid":1,"role":"RDB / AOF writing child","date":"2025-01-17T03:47:44.528Z","level":"warning","message":"WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect."}
{"pid":1,"role":"RDB / AOF writing child","date":"2025-01-17T03:47:44.528Z","level":"notice","message":"oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo"}
{"pid":1,"role":"RDB / AOF writing child","date":"2025-01-17T03:47:44.528Z","level":"notice","message":"Redis version=7.4.1, bits=64, commit=00000000, modified=0, pid=1, just started"}
{"pid":1,"role":"RDB / AOF writing child","date":"2025-01-17T03:47:44.528Z","level":"warning","message":"Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf"}
{"pid":1,"role":"master","date":"2025-01-17T03:47:44.528Z","level":"notice","message":"monotonic clock: POSIX clock_gettime"}
{"pid":1,"role":"master","date":"2025-01-17T03:47:44.529Z","level":"notice","message":"Running mode=standalone, port=6379."}
{"pid":1,"role":"master","date":"2025-01-17T03:47:44.529Z","level":"notice","message":"Server initialized"}
{"pid":1,"role":"master","date":"2025-01-17T03:47:44.529Z","level":"notice","message":"Ready to accept connections tcp"}
{"pid":1,"role":"master","date":"2025-01-17T03:47:45.341Z","level":"notice","message":"User requested shutdown..."}
{"pid":1,"role":"master","date":"2025-01-17T03:47:45.341Z","level":"notice","message":"Saving the final RDB snapshot before exiting."}
{"pid":1,"role":"master","date":"2025-01-17T03:47:45.345Z","level":"notice","message":"DB saved on disk"}
{"pid":1,"role":"master","date":"2025-01-17T03:47:45.345Z","level":"warning","message":"Redis is now ready to exit, bye bye..."}

P.S. I adapted using @zuiderkwast comment: https://github.com/redis/redis/issues/12918#issuecomment-1880735488. You should be able to adapt it to your specific needs.

Hope this helps (although official JSON output would be more desirable).