https://dagster.io/ logo
#deployment-kubernetes
Title
# deployment-kubernetes
k

Ken

06/14/2020, 10:28 AM
hello guys, Thanks for the great work on 0.8.0, I am seeing many things that I really like. Regarding setting the resources for the K8S Job, I am having difficulty using it for my pipeline. 1. Context: • I am using the new feature dagster-celery-k8s job executor. • My pipeline creates a master GKE job -> Submits task-run requests to message-queue -> Celery picks up and dispatches GKE job runs for each solid. ( afaik ) 2. Problem: It seems that when GKE Master Job submits the task-run, Kombu is used to serialize the request payload. with 0.8.0, task-run payload now contains the
V1ResourceRequirements
Object. [1] This object is not JSON-serializable => Kombu fails => Pipeline launch fails. Reference: [1]: /dagster/python_modules/libraries/dagster-celery/dagster_celery -> calling
<http://task.si|task.si>()
=> passing in
resources=resources
What do you think about this error ?
n

nate

06/14/2020, 2:44 PM
hey Ken I think this is a bug we introduced in 0.8.0 - we’ll get a fix together this week
k

Ken

06/14/2020, 8:12 PM
That is awesome, thank you!
c

cat

06/14/2020, 9:50 PM
ah sorry, fixing!
fix is merged https://dagster.phacility.com/R1:9956e307c6db. thanks for reporting!
will be out in the next release, eta June 18 today
🎉 1
k

Ken

06/15/2020, 11:49 PM
Thank you Cat!