Ability to force use of external-logs feature

[Fabian-K]

Feature request

It should be possible to force the use of an external-logs service. I think it is currently only used when logs are no longer available via the k8s API.

Use case

In our case, the pipeline emits structured logs in high granularity. Some of it is relevant for the enduser, however, other parts are only required for internal support. We want to hide those to not confuse the enduser. Therefore our implementation of external-logs would apply the relevant filtering.

Thanks,
Fabian

[AlanGreene]

Thanks for the feature request @Fabian-K. Can you share more about the approach you've taken with the structured logs?

We have an upcoming feature to allow filtering log levels in the client, using GitHub Actions workflow command-style logs. e.g. ::debug::something the user probably doesn't need to see, and have considered potentially allowing the definition of custom formats in future.

I'd love to hear more about the types of logs you're producing to see if there's some way we could accommodate that as we evolve the log viewer in the Dashboard to provide a better experience for everyone.

[AlanGreene]

As for the ability to force use of external logs, that's something we could investigate and might consider for a future release.

Does your external log implementation serve partial logs while the step is running or does it just kick in after the run is complete? Does it support streaming the response?

[Fabian-K]

Thanks for the quick reply!

Logs are in JSON format and always have common fields such as timestamp, logger, level & message. We can however add additional values to it.

{
  "@timestamp": "2024-11-28T14:19:04.096709352Z",
  "host.name": "pod-79685bb579-mg49m",
  "log.logger": "MyLogger",
  "log.level": "INFO",
  "message": "Some log message",
  "labels": {
    "context": "DefaultDispatcher-worker-8"
  },
  "user-facing": "true"
}

We would probably start with something easy for the filtering such as "only show LEVEL >= INFO" but might end with "only show user-facing = true" if we see benefits. As users don´t want to read JSON, our service would also transform it to sth readable like: [MyLogger] Some log message.

To have a good user experience, we would support serving partial logs including streaming in our implementation. If other implementations don´t want to support this, the adjustments could consider the value for --stream-logs

[AlanGreene]

Thanks for the extra detail, that's really helpful.