Use Ignore_Older tail parameter to prevent "Too many files open" error

[DmitryD13]

Hello everyone,

we have some Windows servers in our system where fluent-bit service has to monitor quite a lot of logfiles from different applications. To avoid the "Too many files open" error, which we had in the past, we consider, among other things, to use the Ignore_Older parameter of the Tail input plugin. Now, after reading the documentation and testing the behaviour locally, I am confused with regard to the two following points:

1. Correct/final meaning of this parameter? There are several fluent-bit versions, where the documentation states, that this parameter relates to the lines in a logfile (meaning, that the files will be always read and only after processing them, it is decided, whether the current line will be sent to the output). There are also versions, where the parameter relates to the whole file (or better to say its modification timestamp).
2. With regard to the variant with the whole file, how does fluent-bit process technically? Does it check the modification timestamp and stops, if it is "older", without even opening the file or does it open the file, read the data, then check the modification timestamp and just not send the data to the configured output, if the modification timestamp is "older"? While testing locally, I had the logging level set to "trace", but, unfortunatelly, I could not say, which from the both variants above applied.

I hope, that I can get some answers here. I am also open to other suggestions, how to prevent the Tail input plugin from opening/reading too many logfiles.

Kind regards,
Dmitry