Feature request: Error message templating

[adrienyhuel]

WithErrorCallback() callback print error log messages with predefined Modsecurity format.

But this format lacks some informations to be ingested in our log sink (actually ElasticSearch) and be analyzed by developers :

• real client ip behind a reverse proxy : could be fixed by feat: ✨ Add caddy client_ip variable parsing #184
• server name : ServerName is stored in transaction, but not exploited in error log format
• maybe some custom headers
• ...

If we could template the error log message with caddy replacer, we could add any relevant request information to log, and then add it in our log sink to analyze it.

@fzipi : what do you think about this ?
I can make a PR if needed, I already have a working branch on my fork

[fzipi]

Hmmm... what about sending the audit log instead to elastic?

[adrienyhuel]

Hmmm... what about sending the audit log instead to elastic?

I thought about it, but the audit log is managed directly by coraza, and we can't configure it through Caddy (log rotation, compression...)
On VM it could work, but require additional config (linux logrotate), but not in container.

And there is no option to pass a custom logger to coraza when creating the WAF instance

[fzipi]

Hmmm... how do you send your logs? Writing to file or stdout? If you write to a file, then you can just use filebeat to ship and rotate...

[adrienyhuel]

Actually we use in VM :
Caddy -> log file -> rsyslogd -> fluentd -> Elastic

@fzipi
I just tried to activate audit log :

• Native format is not readable, and way too difficult to parse by Fluentd (or others like FileBeat, Logstash...)
• JSON format may be easily parseable by fluentd to only send some fields to Elastic (like host, method, uri, and a message with all rules message concatened)

[fzipi]

I've been using native audit log with filebeat since more than 6 years ago. Works like a charm in hundreds of servers. That was using modsecurity. But should be the same in this case.

[adrienyhuel]

@fzipi

Native format logs are multiline, and rsyslog doesn't work well with multiline (it needs a regex to detect start of message)
Moreover, it seems that Coraza remove the A and Z parts (which prints lines for start and end of message) here : https://github.com/corazawaf/coraza/blob/e4f82ad17c5b52b909e21bab29c88f7c11c3bdde/types/waf.go#L133
So in Native format, those lines can't be used to check the start and end of message

I managed to parse JSON Audit logs with Fluent to have a message that suits our needs in Elastic (it's mostly just a field mapping) :

{
   "host":"api.xxxx.fr",
   "client_ip":"XX.XX.XX.XX",
   "method":"POST",
   "uri":"/api?id=<script>alert(123)</script>",
   "blocking":true,
   "message":"POST without Content-Length or Transfer-Encoding headers: 0\nXSS Attack Detected via libinjection: Matched Data: XSS data found within ARGS:id: <script>alert(123)</script>\nXSS Filter - Category 1: Script Tag Vector: Matched Data: <script> found within ARGS:id: <script>alert(123)</script>\nNoScript XSS InjectionChecker: HTML Injection: Matched Data: <script found within ARGS:id: <script>alert(123)</script>\nJavascript method detected: Matched Data: alert( found within ARGS:id: <script>alert(123)</script>\nInbound Anomaly Score Exceeded (Total Score: 23): "
}

Last thing I have to do now is to configure logrotate, maybe you have a example ? I read in an issue that I should use copytruncate.

Anyway, thanks for the help 👍

[adrienyhuel]

I played a bit with audit logs options, to achieve exactly what we wanted.

What we wanted :
Log only errors (ie : when some rules are matched), in any SecRuleEngine mode (On or DetectionOnly), to be able to see when request are blocked, or would be blocked if engine was "On".

It appears that :

• SecAuditEngine On -> logs all requests, even those who don't match any rules
• SecAuditEngine RelevantOnly + SecAuditLogRelevantStatus ".*" print logs only when :
  • transaction is interrupted with any status (SecRuleEngine On + deny, drop, redirect)
  • response status from backend is any, and at least one matching rule had "auditlog" enabled

It's not easy to understand, but coraza is very modular :)

[fzipi]

Returning 403 is a defacto standard, but not the only choice. One thing that I do is to play with the return error from the WAF and then use a custom response.

Disclaimer: _I didn't do this using Coraza + Caddy, but the conceptual idea is applicable IMHO.

Example:

• the WAF will return a deny action.
• deny can be set to anything, e.g. error code 418, 451, or 499.
• then the web server catches it and shows a custom error page (using https://caddyserver.com/docs/caddyfile/directives/handle_errors)
• custom error page is a themed page tailored to your site/virtual host (better for UX), and also you can add a custom error message: "Your transaction was blocked. If you think this is an error, use code coraza transaction ID as reference and contact your@soc.address". You might need execution or templating here, don't know 100% how to do it but the transaction id must be passed to the template (I've done this both with nginx/apache using different options. SSI, lua, you name it).
• This page catches error 418|451, and instead returns 403.

Result:

• you get only the WAF logged transactions, based on code 418|451|499 (or the one you choose) as you use the same code as relevant log status
• you get a better UX if there is an actual user and it was a false positive. User can contact you with a precise ID, so you can look at logs.

Hope this is useful. If you get it working, I think it is a good recipe for others and we could add it in our website 😄

[adrienyhuel]

Actually we only expose APIs with Caddy, for a mobile app.
We only want an empty 403 response if the WAF deny the request, as we should not expose error details to a potential attacker.

handle_errors is a good thing in Caddy, but it can't tell where the error comes from (ie, if we have several handlers which can throw a 403 error by example)

So for now I'm happy with the default response from WAF, and auditing all transactions who matched a rule.

[adrienyhuel]

All is working OK now :

• Audit logs in JSON format, all informations needed inside (upgrading coraza to 3.2.2 to provide the "is_interrupted" field, to log if the transaction was blocked or not)
• Logrotate working good with postrotate script reloading Caddy to make it write in new log files
• handle_errors in Caddy config to return a 403 response, regardless of the status code in Coraza rule.