[rsyslog] rsyslog limited to 2000 messages per second?How would i boost it to 1, 000, 000?

[senaps via rsyslog]

Hi everybody.

i am using centos 7 and syslog to log some activity for my application.
unfortunately it seems that i am generating a very large amount of logs
that need to be written to syslog, but most of these messages ate lost.

i have tested different things, but now, i have a pythonic code like this:

c = 0
while True:
syslog(str(c) + "messages saved" )
c += 1
print c

in code output, i get about 1,000,000 messages printed, but while im
watching the messages file, i only get 2000 and everything hangs. i have no
more messages. messing with the config file, i was able to boost it to 2000
messages, some hang time and then 2000 more messages( the counter would say
100,000'th message) witch is still a very very large data lost.

here is the config file that i currently have:




#### MODULES ####

# The imjournal module bellow is now used as a message source instead of
imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via
logger command)
$SystemLogRateLimitInterval 0
$SystemLogRateLimitBurst 0

$ModLoad imjournal # provides access to the systemd journal
$SystemLogRateLimitInterval 0
$SystemLogRateLimitBurst 0


#### RULES ####
*.* /var/log/messages





and this is part of the message that i get.

Aug 17 11:13:21 HP-Test Avapot[16635]: 782044-----------
Aug 17 11:13:21 HP-Test Avapot[16635]: 782044-----------
Aug 17 11:13:50 HP-Test journal: Suppressed 722551 messages from
/user.slice/user-0.slice
Aug 17 11:13:50 HP-Test journal: Suppressed 722551 messages from
/user.slice/user-0.slice


is there anyway to fix this problem? i need this amount of load really.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

[David Lang]

The first thing I would do is check if the systemd journal is the limiting
factor (which is what I expect), either by bypassing it and going directly to
rsyslog, or my checking to see how many logs you have in the journal.

I suspect that you are running into journal limits.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

[Thomas Deutschmann via rsyslog]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Post by senaps via rsyslog
Aug 17 11:13:50 HP-Test journal: Suppressed 722551 messages from
/user.slice/user-0.slice

Your journald is rate limiting. Check your journald config file
(typically /etc/systemd/journald.conf) for RateLimitInterval and
RateLimitBurst.

Like David already said: For best performance try to avoid/bypass
systemd/journald completely. Remember that journald has to maintain a
database... if you are processing logs with rsyslog, especially on a
centralized syslog server, you don't want double processing.


- --
Regards,
Thomas

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0

iQJ8BAEBCgBmBQJZlijqXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQzM0M1ODQ4MkM0MDIyOTJEMkUzQzVDMDY5
NzA5RjkwQzNDOTZGRkM4AAoJEJcJ+Qw8lv/ICI8P/04q9QaSn+GUoPs9Lj3bYFq0
hUnSwjJXvOlZ6aufMZhVwlxgTVHvlx4dJwSRTc314dyLsMLlAQHCRc6m/yNk+AEs
Klbqdj5Tx/zD9ZcUutbpPpu6CZEkfgD8AUOeT7GdnGy8cNlMsyJSzOplwmcBMuFm
juNGFQGlzZQdc5rTqOLdaaRS0+9p1T64UXetPltLUSTvxN8WOMiejltE1b2+jJ/l
saNTsAbgMqO8rwIr0caBnH6CmbgwJA68aUOLM9zaROmcG9hMGL3OXzxZchkFf0Av
AAhguPYlOEoanJ8hpMkyE+4qEska3n8IBRHMuHGm990wloCazJRiKOogxV1PHRHu
7r11gLkgv+yOPsFSHOAm1U3YVk+f/cc07CJVQDHHy3Rxdjrplr4VOOaf9d4PqODA
98DSwwT9LtBQ5Jz19oMYDY/Vxzfsh4gO4cBsEWbe3vFwjWtjs2ix1pJ5iQwjeL/1
2HZQZMI3m04fyAK+gmzdfmKBph5oMgmBKoH5kJK/yadX3XjkPJ8yOuc6zkH+e/3O
wAgMCSBEZXWjVQMPSTQCiunX9iQ2yyAnkQ4AC5gRS7LJPr82MG2uSjOeo2CWXVMp
p/wQhUb1Lg8Y4Bdg7g9YhlsR5YNfZupspRyPXfaLZkxiTGUhLY/VH3vh/am44LTU
O0SRck88MJ30VMCNx6DV
=hDqk
-----END PGP SIGNATURE-----
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.