[rsyslog] It's not necessary to separate JSON field with space?

[eSX via rsyslog]

Hi all

When the log message part is JSON, a message processing like:
parse json with mmjsonparser,
process variables in $!.
unset variables, e.g. unset $!foo!bar;

Alright, send a processed message to next action.
But there were some variables unset, I don't want them to appear in next
action.
So I have to format the message from $! but not $msg.
like:
template (name = "next_action_msg" type = "string" string = "%$!%\n")

There's nothing wrong with it, but a bit flaw.
The fields in message from $! was separated by SPACE. like:
{"foo": {"bar": "hello"}}
but not:
{"foo":{"bar":"hello"}}

A space separated json message could waste more disk space and DB.
It's acceptable, but not necessary for machine readable message.
For human readable, extra spaces added is not enough also, pretty-formated
messages are needed.

There is a low cost improvement method to avoid it.
The json is converted to string in msg.c:getJSONPropVal(), by
calling json_object_get_string().
json_object_get_string() will add space automaticlly, by calling
json_object_to_json_string_ext() with flag JSON_C_TO_STRING_SPACED.
json_object_to_json_string_ext() can accept more flags,

json_object_to_json_string_ext(field, JSON_C_TO_STRING_SPACED) ==
json_object_get_string(field)

json_object_to_json_string_ext(field, JSON_C_TO_STRING_PLAIN) means no more
extra space added.

Any ideas?

Thanks

[Rich Megginson via rsyslog]

Post by eSX via rsyslog
Hi all
parse json with mmjsonparser,
process variables in $!.
unset variables, e.g. unset $!foo!bar;
Alright, send a processed message to next action.
But there were some variables unset, I don't want them to appear in next
action.
So I have to format the message from $! but not $msg.
template (name = "next_action_msg" type = "string" string = "%$!%\n")
There's nothing wrong with it, but a bit flaw.
{"foo": {"bar": "hello"}}
{"foo":{"bar":"hello"}}
A space separated json message could waste more disk space and DB.
It's acceptable, but not necessary for machine readable message.
For human readable, extra spaces added is not enough also, pretty-formated
messages are needed.
There is a low cost improvement method to avoid it.
The json is converted to string in msg.c:getJSONPropVal(), by
calling json_object_get_string().
json_object_get_string() will add space automaticlly, by calling
json_object_to_json_string_ext() with flag JSON_C_TO_STRING_SPACED.
json_object_to_json_string_ext() can accept more flags,
json_object_to_json_string_ext(field, JSON_C_TO_STRING_SPACED) ==
json_object_get_string(field)
json_object_to_json_string_ext(field, JSON_C_TO_STRING_PLAIN) means no more
extra space added.
Any ideas?

Looks like you already have the coding part figured out.  The one catch
is that the current behavior is to add the space.  Even though a valid
JSON parser should accept a space or not, I guarantee you that if you
remove the space, it will break someone's application that is
non-standard.  So, to mitigate this situation, you will need to add a
flag to the JSON formatter to allow you to specify plain or spaced, with
the default value being spaced.

My suggestion is that you submit a PR to implement this, including a
docs PR to document the new flag, and add a test for the JSON parser to
test both spaced and plain.

Post by eSX via rsyslog
Thanks

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you D

[eSX via rsyslog]

I very much agree with this.
There must be a lot of non-standard implementations for log parser.
A flag/swith would be a good choice.
I will submit PR if I have time. ;)


Rich Megginson via rsyslog <***@lists.adiscon.com> 于2018年8月7日周二
下午9:55写道：

Post by eSX via rsyslog

Post by eSX via rsyslog
Hi all
parse json with mmjsonparser,
process variables in $!.
unset variables, e.g. unset $!foo!bar;
Alright, send a processed message to next action.
But there were some variables unset, I don't want them to appear in next
action.
So I have to format the message from $! but not $msg.
template (name = "next_action_msg" type = "string" string = "%$!%\n")
There's nothing wrong with it, but a bit flaw.
{"foo": {"bar": "hello"}}
{"foo":{"bar":"hello"}}
A space separated json message could waste more disk space and DB.
It's acceptable, but not necessary for machine readable message.
For human readable, extra spaces added is not enough also,

pretty-formated

Post by eSX via rsyslog
messages are needed.
There is a low cost improvement method to avoid it.
The json is converted to string in msg.c:getJSONPropVal(), by
calling json_object_get_string().
json_object_get_string() will add space automaticlly, by calling
json_object_to_json_string_ext() with flag JSON_C_TO_STRING_SPACED.
json_object_to_json_string_ext() can accept more flags,
json_object_to_json_string_ext(field, JSON_C_TO_STRING_SPACED) ==
json_object_get_string(field)
json_object_to_json_string_ext(field, JSON_C_TO_STRING_PLAIN) means no

more

Post by eSX via rsyslog
extra space added.
Any ideas?

Looks like you already have the coding part figured out. The one catch
is that the current behavior is to add the space. Even though a valid
JSON parser should accept a space or not, I guarantee you that if you
remove the space, it will break someone's application that is
non-standard. So, to mitigate this situation, you will need to add a
flag to the JSON formatter to allow you to specify plain or spaced, with
the default value being spaced.
My suggestion is that you submit a PR to implement this, including a
docs PR to document the new flag, and add a test for the JSON parser to
test both spaced and plain.

Post by eSX via rsyslog
Thanks

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

[Rich Megginson via rsyslog]

If you are unable to submit a PR, then please submit an issue tracker.

Post by eSX via rsyslog
I very much agree with this.
There must be a lot of non-standard implementations for log parser.
A flag/swith would be a good choice.
I will submit PR if I have time. ;)

Post by eSX via rsyslog
Hi all
parse json with mmjsonparser,
process variables in $!.
unset variables, e.g. unset $!foo!bar;
Alright, send a processed message to next action.
But there were some variables unset, I don't want them to appear

in next

Post by eSX via rsyslog
action.
So I have to format the message from $! but not $msg.
template (name = "next_action_msg" type = "string" string =

"%$!%\n")

Post by eSX via rsyslog
There's nothing wrong with it, but a bit flaw.
{"foo": {"bar": "hello"}}
{"foo":{"bar":"hello"}}
A space separated json message could waste more disk space and DB.
It's acceptable, but not necessary for machine readable message.
For human readable, extra spaces added is not enough also,

pretty-formated

Post by eSX via rsyslog
messages are needed.
There is a low cost improvement method to avoid it.
The json is converted to string in msg.c:getJSONPropVal(), by
calling json_object_get_string().
json_object_get_string() will add space automaticlly, by calling
json_object_to_json_string_ext() with flag JSON_C_TO_STRING_SPACED.
json_object_to_json_string_ext() can accept more flags,
json_object_to_json_string_ext(field, JSON_C_TO_STRING_SPACED) ==
json_object_get_string(field)
json_object_to_json_string_ext(field, JSON_C_TO_STRING_PLAIN)

means no more

Post by eSX via rsyslog
extra space added.
Any ideas?

Looks like you already have the coding part figured out.  The one catch
is that the current behavior is to add the space.  Even though a valid
JSON parser should accept a space or not, I guarantee you that if you
remove the space, it will break someone's application that is
non-standard.  So, to mitigate this situation, you will need to add a
flag to the JSON formatter to allow you to specify plain or spaced, with
the default value being spaced.
My suggestion is that you submit a PR to implement this, including a
docs PR to document the new flag, and add a test for the JSON parser to
test both spaced and plain.

Post by eSX via rsyslog
Thanks

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
--
eSX

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and

[eSX via rsyslog]

OK,
here is my issues:
https://github.com/rsyslog/rsyslog/issues/2913

Post by Rich Megginson via rsyslog
If you are unable to submit a PR, then please submit an issue tracker.

Post by eSX via rsyslog
I very much agree with this.
There must be a lot of non-standard implementations for log parser.
A flag/swith would be a good choice.
I will submit PR if I have time. ;)

Post by eSX via rsyslog
Hi all
parse json with mmjsonparser,
process variables in $!.
unset variables, e.g. unset $!foo!bar;
Alright, send a processed message to next action.
But there were some variables unset, I don't want them to appear

in next

Post by eSX via rsyslog
action.
So I have to format the message from $! but not $msg.
template (name = "next_action_msg" type = "string" string =

"%$!%\n")

Post by eSX via rsyslog
There's nothing wrong with it, but a bit flaw.
{"foo": {"bar": "hello"}}
{"foo":{"bar":"hello"}}
A space separated json message could waste more disk space and DB.
It's acceptable, but not necessary for machine readable message.
For human readable, extra spaces added is not enough also,

pretty-formated

Post by eSX via rsyslog
messages are needed.
There is a low cost improvement method to avoid it.
The json is converted to string in msg.c:getJSONPropVal(), by
calling json_object_get_string().
json_object_get_string() will add space automaticlly, by calling
json_object_to_json_string_ext() with flag JSON_C_TO_STRING_SPACED.
json_object_to_json_string_ext() can accept more flags,
json_object_to_json_string_ext(field, JSON_C_TO_STRING_SPACED) ==
json_object_get_string(field)
json_object_to_json_string_ext(field, JSON_C_TO_STRING_PLAIN)

means no more

Post by eSX via rsyslog
extra space added.
Any ideas?

Looks like you already have the coding part figured out. The one catch
is that the current behavior is to add the space. Even though a valid
JSON parser should accept a space or not, I guarantee you that if you
remove the space, it will break someone's application that is
non-standard. So, to mitigate this situation, you will need to add a
flag to the JSON formatter to allow you to specify plain or spaced, with
the default value being spaced.
My suggestion is that you submit a PR to implement this, including a
docs PR to document the new flag, and add a test for the JSON parser to
test both spaced and plain.

Post by eSX via rsyslog
Thanks

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
--
eSX