Discussion:
[rsyslog] TCP Keepalive on client and server side
Peter Viskup via rsyslog
2018-10-11 10:12:40 UTC
Permalink
From my latest observation it seems the TCP Keepalive is not working as
expected in our environment. We do run rsyslog 8.15, which I know is old,
but cannot update.

Want to make sure how the TCP Keepalive is developed in rsyslog and whether
there were some changes since 8.15 release. At the moment not interested in
deployment of RELP.

My understanding from reading TCP-Keepalive-HOWTO
<http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html> is both sides can
send empty ACK packet and the other side should response with empty ACK.
That way the TCP session will remain open and all the network devices will
keep track of it.

In our environment the rsyslog server send ACK with no data, but client
does not reply. After some time the rsyslog server closes the connection
sending TCP RST packet.
From the client side the rsyslog tries to open new TCP connection. Once
opened, the client just closes it. It seems to me like client's TCP
Keepalive.

Why the client does not sent ACK reply to server's keepalive?
Is the client's TCP session opening the proper way to implement keepaliving
on client side?
--
Peter
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Rainer Gerhards
2018-10-11 10:57:47 UTC
Permalink
I did a quick search through the ChangeLog

https://github.com/rsyslog/rsyslog/blob/master/ChangeLog

and it shows ample changes since Dec 2015 (8.15). Have a look yourself.

HTH
Rainer
El jue., 11 oct. 2018 a las 12:12, Peter Viskup via rsyslog
Post by Peter Viskup via rsyslog
From my latest observation it seems the TCP Keepalive is not working as
expected in our environment. We do run rsyslog 8.15, which I know is old,
but cannot update.
Want to make sure how the TCP Keepalive is developed in rsyslog and whether
there were some changes since 8.15 release. At the moment not interested in
deployment of RELP.
My understanding from reading TCP-Keepalive-HOWTO
<http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html> is both sides can
send empty ACK packet and the other side should response with empty ACK.
That way the TCP session will remain open and all the network devices will
keep track of it.
In our environment the rsyslog server send ACK with no data, but client
does not reply. After some time the rsyslog server closes the connection
sending TCP RST packet.
From the client side the rsyslog tries to open new TCP connection. Once
opened, the client just closes it. It seems to me like client's TCP
Keepalive.
Why the client does not sent ACK reply to server's keepalive?
Is the client's TCP session opening the proper way to implement keepaliving
on client side?
--
Peter
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
Peter Viskup via rsyslog
2018-10-11 13:32:33 UTC
Permalink
Thank you Rainer,
the Changelog answered why client is not answering keepalive packets
(bug fixed in 8.18).

What about the TCP session open&close on client side?
This happen every 16 seconds in parallel with other TCP session opened
and used for data transfer.

Following is session export from pcap:
"5606","316.433579","client","server","TCP","74","0x4da5
(19877)","51112 → 1514 [SYN] Seq=0 Win=29200 Len=0 MSS=1460
SACK_PERM=1 TSval=2136435442 TSecr=0 WS=128"
"5612","316.702389","server","client","TCP","74","0x0000 (0)","1514 →
51112 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1350 SACK_PERM=1
TSval=339748485 TSecr=2136435442 WS=128"
"5613","316.702472","client","server","TCP","66","0x4da6
(19878)","51112 → 1514 [ACK] Seq=1 Ack=1 Win=29312 Len=0
TSval=2136435509 TSecr=339748485"
"5614","316.702562","client","server","TCP","66","0x4da7
(19879)","51112 → 1514 [FIN, ACK] Seq=1 Ack=1 Win=29312 Len=0
TSval=2136435509 TSecr=339748485"
"5615","316.970442","server","client","TCP","66","0x39d7
(14807)","1514 → 51112 [FIN, ACK] Seq=1 Ack=2 Win=29056 Len=0
TSval=339748552 TSecr=2136435509"
"5616","316.970503","client","server","TCP","66","0x4da8
(19880)","51112 → 1514 [ACK] Seq=2 Ack=2 Win=29312 Len=0
TSval=2136435576 TSecr=339748552"

There are no data transferred.

Peter

On Thu, Oct 11, 2018 at 12:58 PM Rainer Gerhards
Post by Rainer Gerhards
I did a quick search through the ChangeLog
https://github.com/rsyslog/rsyslog/blob/master/ChangeLog
and it shows ample changes since Dec 2015 (8.15). Have a look yourself.
HTH
Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT P
Rainer Gerhards
2018-10-16 07:16:28 UTC
Permalink
You need to update to the current verson (8.38.0) first before I can
look at this.

Rainer
El jue., 11 oct. 2018 a las 15:32, Peter Viskup
Post by Peter Viskup via rsyslog
Thank you Rainer,
the Changelog answered why client is not answering keepalive packets
(bug fixed in 8.18).
What about the TCP session open&close on client side?
This happen every 16 seconds in parallel with other TCP session opened
and used for data transfer.
"5606","316.433579","client","server","TCP","74","0x4da5
(19877)","51112 → 1514 [SYN] Seq=0 Win=29200 Len=0 MSS=1460
SACK_PERM=1 TSval=2136435442 TSecr=0 WS=128"
"5612","316.702389","server","client","TCP","74","0x0000 (0)","1514 →
51112 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1350 SACK_PERM=1
TSval=339748485 TSecr=2136435442 WS=128"
"5613","316.702472","client","server","TCP","66","0x4da6
(19878)","51112 → 1514 [ACK] Seq=1 Ack=1 Win=29312 Len=0
TSval=2136435509 TSecr=339748485"
"5614","316.702562","client","server","TCP","66","0x4da7
(19879)","51112 → 1514 [FIN, ACK] Seq=1 Ack=1 Win=29312 Len=0
TSval=2136435509 TSecr=339748485"
"5615","316.970442","server","client","TCP","66","0x39d7
(14807)","1514 → 51112 [FIN, ACK] Seq=1 Ack=2 Win=29056 Len=0
TSval=339748552 TSecr=2136435509"
"5616","316.970503","client","server","TCP","66","0x4da8
(19880)","51112 → 1514 [ACK] Seq=2 Ack=2 Win=29312 Len=0
TSval=2136435576 TSecr=339748552"
There are no data transferred.
Peter
On Thu, Oct 11, 2018 at 12:58 PM Rainer Gerhards
Post by Rainer Gerhards
I did a quick search through the ChangeLog
https://github.com/rsyslog/rsyslog/blob/master/ChangeLog
and it shows ample changes since Dec 2015 (8.15). Have a look yourself.
HTH
Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PL
Peter Viskup via rsyslog
2018-10-18 16:08:37 UTC
Permalink
Ack,
will check after upgrade.

As an workaround the tcp_retries2 kernel option was lowered according
https://pracucci.com/linux-tcp-rto-min-max-and-tcp-retries2.html
This make us sure the TCP forward session will be recognized as broken
sooner than default 924 seconds. We are loosing messages waiting in
TCP buffer on client side, but accepted.
--
Peter
On Tue, Oct 16, 2018 at 9:17 AM Rainer Gerhards
Post by Rainer Gerhards
You need to update to the current verson (8.38.0) first before I can
look at this.
Rainer
El jue., 11 oct. 2018 a las 15:32, Peter Viskup
Post by Peter Viskup via rsyslog
Thank you Rainer,
the Changelog answered why client is not answering keepalive packets
(bug fixed in 8.18).
What about the TCP session open&close on client side?
This happen every 16 seconds in parallel with other TCP session opened
and used for data transfer.
"5606","316.433579","client","server","TCP","74","0x4da5
(19877)","51112 → 1514 [SYN] Seq=0 Win=29200 Len=0 MSS=1460
SACK_PERM=1 TSval=2136435442 TSecr=0 WS=128"
"5612","316.702389","server","client","TCP","74","0x0000 (0)","1514 →
51112 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1350 SACK_PERM=1
TSval=339748485 TSecr=2136435442 WS=128"
"5613","316.702472","client","server","TCP","66","0x4da6
(19878)","51112 → 1514 [ACK] Seq=1 Ack=1 Win=29312 Len=0
TSval=2136435509 TSecr=339748485"
"5614","316.702562","client","server","TCP","66","0x4da7
(19879)","51112 → 1514 [FIN, ACK] Seq=1 Ack=1 Win=29312 Len=0
TSval=2136435509 TSecr=339748485"
"5615","316.970442","server","client","TCP","66","0x39d7
(14807)","1514 → 51112 [FIN, ACK] Seq=1 Ack=2 Win=29056 Len=0
TSval=339748552 TSecr=2136435509"
"5616","316.970503","client","server","TCP","66","0x4da8
(19880)","51112 → 1514 [ACK] Seq=2 Ack=2 Win=29312 Len=0
TSval=2136435576 TSecr=339748552"
There are no data transferred.
Peter
On Thu, Oct 11, 2018 at 12:58 PM Rainer Gerhards
Post by Rainer Gerhards
I did a quick search through the ChangeLog
https://github.com/rsyslog/rsyslog/blob/master/ChangeLog
and it shows ample changes since Dec 2015 (8.15). Have a look yourself.
HTH
Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT P
Loading...