How do you tail logs by keeping state in a k8s cluster?

[AkshayLokur]

I am referring to tail - keep_state.

However, I am running fluent-bit in a container on k8s pod. Containers and pods are ephemeral by nature, so how can I store the state of tail input plugin so that it is retained even though pod gets restarted or even a k8s node gets changed for any reason.

A small code snippet for configuring this would be immensely helpful.

Thank you!

[patrick-stephens]

The helm chart has an example - as this is using hostpath then the db can also be kept there.
The other important thing to be clear on is that any pod has the right db mounted, e.g. if your pod is evicted and a new one starts it needs the db from the original and not another pod.

[AkshayLokur]

Thanks @patrick-stephens for your reply!

On a multi-node k8s cluster I have, nodes can get replaced and also data-prepper pods - if restarted - can get placed on other nodes in the same cluster.
Considering this I suspect how much hostPath will be reliable?

[patrick-stephens]

Sure, I was pointing out how it can be done but you have to pick a solution most relevant to your set up. DB file is not large though so you could just monitor it.

The things you need to do are:

1. Provide a DB related to the files you are tailing.
2. Handle pod eviction and rescheduling - the new pod must get the same db as the old pod.
3. Ensure each Fluent Bit "shard" (responsible for specific logs) has a matching DB and this is persistent. DBs are not intended to be shared across all pods - particularly if log filenames overlap.

What files are you tailing?
If a node gets replaced and the files you are tailing are on the node then it doesn't matter if the DB is gone as so are the logs.

[patrick-stephens]

It's a standard persistent question with K8S really, associating state with pods

[carlosrbcunha]

As you are deploying normally as a DaemonSet, if a pod crashes or gets deleted, another pod with spawn in the same node and will pick up the work where the other left off.
We are using the same hostPath for the whole DaemonSet (one per node) to keep the tail DB but also to handle some disk back pressure in case of connection problems or just too many logs to be sent at any given time. This last situation can lead to some issues if the node capacity is too low or if your log stream is too intense. Node gets low on FS and pods start to be evicted. This can cause the very verbose app to get evicted and then fluent-bit recovers but you need to test this into your setup.