feat(fluent-bit): set system-node-critical as a default priorityClass

[m00lecule]

Issues:
#54

In Kubernetes there is a good practice to specify a system-node-critical priority class for monitoring agents running as daemonsets.

ref: https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[m00lecule]

@lisguo may I ask for a review?

[m00lecule]

@lisguo May I ask for a review again? The Integration tests are failing due to Error: Credentials could not be loaded, please check your action inputs: Could not load credentials from any providers which doesn't seem to be connected with my changes.

[markandersontrocme]

@m00lecule we should also add the priorityClass to the cloudwatch-agent? https://github.com/aws-observability/helm-charts/blob/main/charts/amazon-cloudwatch-observability/templates/linux/cloudwatch-agent-daemonset.yaml#L23

[m00lecule]

@m00lecule we should also add the priorityClass to the cloudwatch-agent? https://github.com/aws-observability/helm-charts/blob/main/charts/amazon-cloudwatch-observability/templates/linux/cloudwatch-agent-daemonset.yaml#L23

@markandersontrocme I believe so, the general principle is to ensure that monitoring agents are running on all EKS nodes

[markandersontrocme]

@markandersontrocme I believe so, the general principle is to ensure that monitoring agents are running on all EKS nodes

I fee like it would make sense to add that to this PR 🙏

[milldr]

@m00lecule thanks for opening this PR! I'm hitting this same issue for both fluent-bit and cloudwatch-agent. Is there anything we can do to push this through?

[voidlily]

I've been manually working around this by setting the priorityClassName on the deployments/daemonsets, but that obviously doesn't scale well for version updates or deploying to multiple clusters. This PR would be useful to me as well (also adding system-cluster-critical, possibly system-node-critical if it's a daemonset) for the cloudwatch agent.

See also the containers-roadmap ticket I linked earlier in this thread

[m00lecule]

@milldr @voidlily lets keep fingers crossed together that somebody from AWS (@lisguo) will review proposed changes.

[m00lecule]

@milldr @markandersontrocme priorityClassName support to cloudwatch-agent daemonset has been introduced here 9e60aaa