[rsyslog] Delete log files after sending logs to Elasticsearch

Discussion:

Alec Swan

9 years ago

Hello,

I am using rsyslog to read log files and send logs to Elasticsearch. I am
looking for ideas on how to delete log files after their content is
successfully sent to Elasticsearch.

Thanks,

Alec
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

Radu Gheorghe

9 years ago

Permalink

Hi Alec,

A dirty one would be to have a cron job that:
- looks at the last modified date of a file
- queries Elasticsearch by timestamp. To see if the latest log from
that file got in. If it did (and maybe also if the Elasticsearch
snapshot completed successfully :p), the file can be deleted

Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

...

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.

m***@gmail.com

9 years ago

Permalink

Working with Elasticsearch you could find many workarounds:

- ES allows you to "fingerprint" messages, so if this value matches
with your log hash(event), you are safe to delete that event

- Using logstash RELP or eve ruby plugin to exec a "delete log line"
after inserting it on ES

- With redis you could use push with queue (and deleting line) and
rpoplpush + cron to check if events are already stored or should be
queued again

- ...

...

Alec Swan

9 years ago

Permalink

Thanks for sharing your thoughts. I am not using Logstash and hence using
fingerprint or logstash plugins are not an option.

Alec

- ES allows you to "fingerprint" messages, so if this value matches with
your log hash(event), you are safe to delete that event
- Using logstash RELP or eve ruby plugin to exec a "delete log line"
after inserting it on ES
- With redis you could use push with queue (and deleting line) and
rpoplpush + cron to check if events are already stored or should be queued
again
- ...
Hi Alec,

Post by Radu Gheorghe
- looks at the last modified date of a file
- queries Elasticsearch by timestamp. To see if the latest log from
that file got in. If it did (and maybe also if the Elasticsearch
snapshot completed successfully :p), the file can be deleted
Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

Post by Alec Swan
Hello,
I am using rsyslog to read log files and send logs to Elasticsearch. I am
looking for ideas on how to delete log files after their content is
successfully sent to Elasticsearch.
Thanks,
Alec
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

...

David Lang

9 years ago

Permalink

Post by Alec Swan
I am using rsyslog to read log files and send logs to Elasticsearch. I am
looking for ideas on how to delete log files after their content is
successfully sent to Elasticsearch.

rsyslog doesn't have any options to remove a file when it's done processing it
(rsyslog assumes that it's reading a file that something else is writing to).

What I would suggest is to use something like logger to read the file and
deliver the lines to rsyslog, then remove the file after it's all been delivered
to rsyslog.

David Lang

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.