[rsyslog] lognorm1 rules with optional message part

[Peter Viskup via rsyslog]

It is for the first time I am working with liblognorm.
Read the documentation for lognorm1, but still not sure how to write
mmnormalize rules for optional parts of syslog message.
The base is RFC5424 message with modified structured-data.

Special SD-ELEMENT [***@123456 relay-ip="timestamp-rfc3339"
...] added to the end of structured-data. Every relay add it's own
relay-ip with timestamp to this element.

On some relay's this SD-ELEMENT needs to be removed. Will this rule
work as expected?

prefix=<%-:number%>%-:number% %-:date-rfc5424% %-:word% %-:word%
%-:number% %-:word%
rule=%orig-sd:string-to:[***@123456 % %time-sd:string-to: ]% %-:rest%

How to reference the 'orig-sd' value in template afterwards?

Is lognorm2 making this easier to implement?

Peter
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

[David Lang]

Post by Peter Viskup via rsyslog
It is for the first time I am working with liblognorm.
Read the documentation for lognorm1, but still not sure how to write
mmnormalize rules for optional parts of syslog message.
The base is RFC5424 message with modified structured-data.
...] added to the end of structured-data. Every relay add it's own
relay-ip with timestamp to this element.

I would suggest not trying to parse this structured data with mmnormalize, let
the rfc5424 parser parse it.

Post by Peter Viskup via rsyslog
On some relay's this SD-ELEMENT needs to be removed. Will this rule
work as expected?
prefix=<%-:number%>%-:number% %-:date-rfc5424% %-:word% %-:word%
%-:number% %-:word%
How to reference the 'orig-sd' value in template afterwards?

log the message with the template RSYSLOG_DebugFormat and you will see the $!
variable tree, with orig-sd under it, you would access it with $!orig-sd

Post by Peter Viskup via rsyslog
Is lognorm2 making this easier to implement?

lognorm2 changes the parsers a little bit and is far more efficient, you really
should use it by default, for backwards compatibility we support the lognorm1
parsers, but that's really just to avoid breaking existing configs.

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

[Peter Viskup via rsyslog]

Post by David Lang

Post by Peter Viskup via rsyslog
...] added to the end of structured-data. Every relay add it's own
relay-ip with timestamp to this element.

I would suggest not trying to parse this structured data with mmnormalize, let
the rfc5424 parser parse it.

The rfc5424 parser provide this string:
STRUCTURED-DATA: '[***@123456
10.x.y.z="2018-11-20T07:55:03.832066+01:00"]'
but I would like to cover the case with other SD-ELEMENTS and removal
only the syslogTimes.
e.g.
STRUCTURED-DATA: '[***@123456 key1="value1"
key2="value2"][***@123456
10.x.y.z="2018-11-20T07:55:03.832066+01:00"]'

Post by David Lang

Post by Peter Viskup via rsyslog
On some relay's this SD-ELEMENT needs to be removed. Will this rule
work as expected?
prefix=<%-:number%>%-:number% %-:date-rfc5424% %-:word% %-:word%
%-:number% %-:word%
How to reference the 'orig-sd' value in template afterwards?

log the message with the template RSYSLOG_DebugFormat and you will see the $!
variable tree, with orig-sd under it, you would access it with $!orig-sd

The rule is not parsing the message as expected. In debug there are
only values in $!:
$!:{ "originalmsg": "pam_unix(sudo:session): session closed for user
root", "unparsed-data": "pam_unix(sudo:session): session closed for
user root" }

rawmsg looks like this (all message parts are parsed by rfc5424
without any issue):
inputname: imtcp rawmsg: '<86>1 2018-11-20T07:55:03.832066+01:00
hostname_10.x.y.z sudo - - [***@123456
10.x.y.z="2018-11-20T07:55:03.832066+01:00"] pam_unix(sudo:session):
session closed for user root'

Peter
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

[David Lang]

Post by Peter Viskup via rsyslog

Post by David Lang

Post by Peter Viskup via rsyslog
...] added to the end of structured-data. Every relay add it's own
relay-ip with timestamp to this element.

I would suggest not trying to parse this structured data with mmnormalize, let
the rfc5424 parser parse it.

10.x.y.z="2018-11-20T07:55:03.832066+01:00"]'
but I would like to cover the case with other SD-ELEMENTS and removal
only the syslogTimes.
e.g.
10.x.y.z="2018-11-20T07:55:03.832066+01:00"]'

rsyslog does not have good tools for dealing with lists.

I will also say that the structured data portion of RFC 5424 is pretty much a
failure, virtually nobody actually uses it. Instead what is being used for
structured data is to make the message portion of the log be JSON. Take the
existing $msg and make it $!msg and then you can add any other variables that
you want (I use $!trusted as the root for my metadata) and then extract them and
manage them individually later.

Post by Peter Viskup via rsyslog

Post by David Lang

Post by Peter Viskup via rsyslog
On some relay's this SD-ELEMENT needs to be removed. Will this rule
work as expected?
prefix=<%-:number%>%-:number% %-:date-rfc5424% %-:word% %-:word%
%-:number% %-:word%
How to reference the 'orig-sd' value in template afterwards?

log the message with the template RSYSLOG_DebugFormat and you will see the $!
variable tree, with orig-sd under it, you would access it with $!orig-sd

The rule is not parsing the message as expected. In debug there are
$!:{ "originalmsg": "pam_unix(sudo:session): session closed for user
root", "unparsed-data": "pam_unix(sudo:session): session closed for
user root" }
rawmsg looks like this (all message parts are parsed by rfc5424
inputname: imtcp rawmsg: '<86>1 2018-11-20T07:55:03.832066+01:00
session closed for user root'

take a look at the log parser tool that liblognorm provides, it includes debug
features so you can see what is happening as you parse. It looks as if you are
leaving the mmnormalize call set to parse the default, which is $msg, not
$rawmsg.

please post your config again (I delete mailing list messages aggressivly, so I
apologize if you posted it recently)

David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.