Hi, I'm evaluating dagster to see if it fits or needs. Right now we mostly need to run k8s containers/pods (usually non-python) with specific commands/args. I found https://docs.dagster.io/_apidocs/libraries/dagster-k8s#ops , but I couldn't find how I can customize the command I need to run based on input arguments given in my job config... any pointers?

also curious to see answers here. I had to do something similar, i ended up doing most of it myself (using kubernetes python api)

hey friends, sorry for the delay here, I think a bunch of the team has been wrapped up in prepping an upcoming release! we’ll get back to you ASAP

that would be great, I really couldn't find a way to do that... Hopefully I missed something, otherwise dagster will unfortunately be out for us...

Hi Felix - the “execute_k8s_job” function here can be called within an op that you create (like any other dagster op) and should allow you to do this: https://docs.dagster.io/_apidocs/libraries/dagster-k8s#dagster_k8s.execute_k8s_job

So the idea behind this is that you write your input/output parsing/passing in a normal Python op and then call execute_k8s_job? Wouldn't that launch more k8s jobs than needed, one for the Python op and one for the job doing the actual work launched via execute_k8s_job. Also would you get the logs, etc from the k8s job launched via execute_k8s_job?

That’s exactly right, yeah. Re: more jobs than needed, the default behavior of dagster if you’re running in k8s is to run all the ops together within a single run pod (there’s a way to run each op in its own pod, but it isn’t enabled by default) - so you shouldn’t get two pods for every op in general, no. Logs from the pod will show up in the “Raw compute logs” tab in dagit - as long as you have configured the compute logs in your helm chart to write to S3 or some other cloud storage engine.

Ok, nice! Will try that... compute logs to S3 is already set up. Just updating the user code is quite cumbersome...