Improve log rotation handling

[brifordwylie]

Right now the Bro log tailing 'kinda' handles log rotation but there are lots of little corner cases that we're not taking care of. We might consider using something like Pygtail (https://github.com/bgreenlee/pygtail). Looking at the project/code they've put a lot of work into handling all those crazy corner cases.

[brifordwylie]

Actually after spending some 'quality time' with Pygtail on another project it doesn't handle some of the use cases we need (rename/create) and the code logic doesn't really support dynamic tailing very well. You can put a wrapper around next that works fine but then weird stuff happens with log rotations not be processed correctly...

[swedishmike]

Just a thought, that might be wide off the mark....

How many users do you think want/need the log tailing? Would it make sense to utilise some other mechanism for the heavy lifting instead?

For example if a heavy/advanced user have a ready Spark/Kafka setup could that be utilised to pull a (more or less) live feed?

As I said, just a thought...

[github-actions]

Stale issue message