I've posted a question regarding async pipeline execution on Q&A https://github.com/dagster-io/dagster/discussions/3342

Aside from

configured()

tricks, there's no way to have a solid that will accept both pipeline values and literals, right?

like

bar(foo(), 1)

in a pipeline

You would have to do

bar.configured(x=1)(foo())

or similar I think?

hi there! I have a small question regarding housekeeping in kubernetes deployments: jobs and pods of completed runs are not being cleaned up at the moment, is this desired? are there any plans to implement some housekeeping mechanism, e.g. to clean such artifacts up after a successful job completion or is it something better taken up by another tool? cheers

Jobs or pods?

Pods should be cleaned up, Jobs generally aren't

Job cleanup is still an alpha feature, https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/

There's some third party options though

https://www.google.com/search?client=firefox-b-1-d&q=kubernetes+job+cleanup will find a bunch 🙂

both jobs and pods are left behind.

which component should clean up pods? maybe I can look into why it's not happening

What state are the pods in?

They should be Succeeded or Failed, not Running

completed

There we go

That's not actually running anywhere 🙂

The API object is tied to the Job via owner ref

so if we clean up jobs, pods are gone too

is this what you're saying?

I think so, check for your version of k8s though

I know back in like 1.15 there wasn't a ref in place so they were handled terribly

you're right, this is what happens indeed. So my question was more like, is there a reason to keep jobs and pods once they are completed?

Like I said, cleanup is still in alpha

As to why, you would have to talk to SIG-APIMachinery, I think it was a concern over the locking required to do it safely causing a negative impact on the controller.

so I'm guessing that if we use some other tooling to clean up jobs & pods, dagster itself wouldn't mind 🙂

thx!

I don't think so, Dagster doesn't access anything K8s-related directly as far as I know, it's just launching the usual worker process machinery inside a Job which then connects into the database and whatnot.

👍 thanks again

Hi all! I'm getting a little stuck with setting up schedules. I expect that this is just some configuration that I'm missing or setting up wrongly... Would be great if somebody who's had some experience with setting this up could help 😌 I'll post the relevant files in this thread - thank you in advance for any pointers!