kubernetes Hi all, wanted to call out a new experimental feature in 0.15.5: the

k8s_job_op

. This Op launches a Kubernetes Job with whatever image and CLI command you provide. Try it out: Docs

This is great @johann. Thank you very much

Can thank Daniel!

does the job spin up the container, poll it for completion, then exit?

Yes, each instance of this op will have a step worker responsible for creating the job and watching it. There’s certainly room for optimization there, if the run worker were to do the creation and polling then 1. you’d eliminate a middle man and 2. it could poll on multiple jobs simultaneously

cool, this sounds really great. i could see myself using this with meltano to house all our dagster workflows in a dagster-specific repo, but to manage the meltano image/dependencies separately. could also see it as a good use case for dbt

This is good, but @Stephen Bailey’s point, if we use it to run meltano, how do we translate that Meltano run as Data Asset ? because at the end of the day, Meltano will have (Source data asset and Target data asset). How does it build a graph of data asset ?