:kubernetes: Hi all, wanted to call out a new expe...
# deployment-kubernetes
kubernetes Hi all, wanted to call out a new experimental feature in 0.15.5: the
. This Op launches a Kubernetes Job with whatever image and CLI command you provide. Try it out: Docs
🎉 2
all the things 2
🙏 3
hmmnod 1
This is great @johann. Thank you very much
Can thank Daniel!
does the job spin up the container, poll it for completion, then exit?
Yes, each instance of this op will have a step worker responsible for creating the job and watching it. There’s certainly room for optimization there, if the run worker were to do the creation and polling then 1. you’d eliminate a middle man and 2. it could poll on multiple jobs simultaneously
👍 1
cool, this sounds really great. i could see myself using this with meltano to house all our dagster workflows in a dagster-specific repo, but to manage the meltano image/dependencies separately. could also see it as a good use case for dbt
This is good, but @Stephen Bailey’s point, if we use it to run meltano, how do we translate that Meltano run as Data Asset ? because at the end of the day, Meltano will have (Source data asset and Target data asset). How does it build a graph of data asset ?