Hi, I'm evaluating dagster to see if it fits or ne...
# ask-community
f
Hi, I'm evaluating dagster to see if it fits or needs. Right now we mostly need to run k8s containers/pods (usually non-python) with specific commands/args. I found https://docs.dagster.io/_apidocs/libraries/dagster-k8s#ops , but I couldn't find how I can customize the command I need to run based on input arguments given in my job config... any pointers?
👍 1
k
also curious to see answers here. I had to do something similar, i ended up doing most of it myself (using kubernetes python api)
d
hey friends, sorry for the delay here, I think a bunch of the team has been wrapped up in prepping an upcoming release! we’ll get back to you ASAP
f
that would be great, I really couldn't find a way to do that... Hopefully I missed something, otherwise dagster will unfortunately be out for us...
d
Hi Felix - the “execute_k8s_job” function here can be called within an op that you create (like any other dagster op) and should allow you to do this: https://docs.dagster.io/_apidocs/libraries/dagster-k8s#dagster_k8s.execute_k8s_job
f
So the idea behind this is that you write your input/output parsing/passing in a normal Python op and then call execute_k8s_job? Wouldn't that launch more k8s jobs than needed, one for the Python op and one for the job doing the actual work launched via execute_k8s_job. Also would you get the logs, etc from the k8s job launched via execute_k8s_job?
d
That’s exactly right, yeah. Re: more jobs than needed, the default behavior of dagster if you’re running in k8s is to run all the ops together within a single run pod (there’s a way to run each op in its own pod, but it isn’t enabled by default) - so you shouldn’t get two pods for every op in general, no. Logs from the pod will show up in the “Raw compute logs” tab in dagit - as long as you have configured the compute logs in your helm chart to write to S3 or some other cloud storage engine.
f
Ok, nice! Will try that... compute logs to S3 is already set up. Just updating the user code is quite cumbersome...