19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#announcements
Title
k
Ken
07/27/2020, 10:57 PMHello Guys, I have a product question:
I see that we have solid_shell, is there any plan for solid_k8s_job ?
So that we can pass [ address of a docker-image + configurations ] to the solid-tags, and that solid will go run a K8S Job using those input ?
Context about why I ask:
My pipeline processes scientific images ( geotiff ), each solid performs a different process ( such as analyze land-temperature, count trees in geotiff, analyze botanical health pattern ... )
The problem I am facing is that I want to add a feature ( 1 new solid ) into this pipeline.
This feature turns a geotiff into map Tiles and upload those tiles to S3 bucket to be used in a web app.
Currently, this new feature requires a huge slur of dependencies ( tilestache, mapnik, cartoCSS binary, nodejs )
Those dependencies are only for this 1 feature, and I don't want to clutter the base-pipeline image.
So I am considering putting that new feature into a separate docker-image, and trigger it as a GKE Job.
s
schrockn
07/27/2020, 11:03 PMno immediate plans to but this is a totally reasonable thing to build
k
Ken
07/27/2020, 11:07 PMThat's cool to know, Thank you Nicholas!
Implementing a custom solid is always a choice, but was wondering if Dagster has anything on the way.
Look forward to seeing it getting made in the future!
s
sashank
07/28/2020, 1:57 AM@Ken if you do end up deciding to implement this yourself, you may find a lot of the code in
dagster-k8sk
Ken
07/28/2020, 3:11 AMIndeed Sashank! Thanks for pointing that out.
I am making heavy use of the launcher code in that library!
s
sashank
07/28/2020, 3:37 PMThat’s great! Please let us know if we can be helpful as you implement it.
If you end up writing something that’s generalizable and you’re interested we can add the solid to
dagster-k8sk
Ken
07/28/2020, 10:33 PMIt's done, I appreciate your thoughts 😄
It was a simple flow of trigger job => poll for completion => print out logs from pod.
I really enjoy the helm chart you guys have too, it helped me plug-and-play env variables quickly. Thanks for the great work on this product!!
I will definitely see if it is generalizable enough, and consider making a PR to add it to dagster-k8s!
s
sashank
07/28/2020, 10:35 PMGlad you were able to implement what you needed fairly quickly. Thanks for the great feedback!