Any advice on using fluentd/bit to upload JSON-formatted application logs from user code in k8s? The standard would be a daemonset, but that pulls from the container's stdout, which is a bit noisy. Our current (on-prem) setup is to have a separate fluentd process that tails a JSON-formatted log file. Is there support for a k8s agent to start multi-container pods for user code (I don't see anything in its Helm values)? Is it a terrible idea to just also run fluentbit in our application image?

you can use these tags to have multi-container pods for jobs: https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#job-or-op-kubernetes-configuration Supporting that for every pod that the agent spins up include the code servers would be the same change that would enable tolerations

Coming back to this, I think you can actually configure a fluentd daemonset to use the tail plugin to read from a file path just fine. Then so long as the fluentd daemonset and the Dagster workspace pod have a shared volume to read/write from/to, that should work smoothly. (Not sure if a local or persistent volume makes a difference.) So I think the only thing needed from Dagster is specifying a volume for the pod via the agent's

workspace.volumes

. I'm not sure how that differs from

@job tags pod_spec_config volumes

, but I like dealing with this in the Helm / kubernetes arena rather than the

@job

arena if they're equivalent.

I think that's right - Dagster doesn't currently provide any functionality for creating additional k8s resources like configmaps for you

For posterity, here's what I got working: • No extra tags on the Dagster

@job

. According to documentation these only affect the k8s run launcher / job per pod, and not the job per step mode which is what I'm using. •

volumes

and

volumeMounts

in the agent's Helm chart. These did add the host volume mount I needed, mapping in

/var/log

to my step pods. So then my application code can write out a JSON-formatted log file that

fluent-bit

can see. •

fluent-bit

as a daemonset. By default this maps

/var/log

into the fluent-bit container, so I didn't need to define extra volumes on that side. And the

fluent-bit

Helm chart has options for its config file which it then publishes as a ConfigMap, so I didn't need to separately set up a ConfigMap. All in all, pretty concise and nicely organized; for the most part the application code remains agnostic of the deployment configuration.