https://dagster.io/ logo
Powered by
s

szalai1

08/13/2020, 12:00 PM
hi, is there any way to run pipelines in a single 
k8s
job using 
celery_k8s_job_executor
? starting a job per solid, in case of small/fast solids makes pipelines slower.
j

johann

08/13/2020, 12:21 PM
Take a look at 
python_modules/libraries/dagster-k8s/dagster_k8s/launcher.py
Currently we only use celery to do each execution step in separate pods, does your use case require celery?
s

szalai1

08/13/2020, 12:27 PM
I think no. based on this : https://docs.dagster.io/overview/run-launchers-executors/executor I thought there is no way to run k8s jobs without celery
j

johann

08/13/2020, 12:30 PM
Gotcha. We have 2 concepts for deployement, run launchers and executors. Run launchers prepare the execution of the entire pipeline, while the executor is what is responsible for running the individual solids
In your case what I think you want is the k8s run launcher (which will kick off a pipeline run in a k8s job), then probably the multiprocess executor (which will run the solids locally in that pod)
s

szalai1

08/13/2020, 12:32 PM
ahh that makes sense, I felt that I'm missing some concept, thank you.
j

johann

08/13/2020, 12:33 PM
You’re welcome! We’re working on making it more clear in the docs, thanks for working through it
🙌 2
I’ll also add you to #k8s-deployment
👍 1
s

szalai1

08/13/2020, 1:06 PM
hmm, I've removed the 
celery_k8s_job_executor
from my mode and set 
multiproc
. but now getting this error: 
dagster.check.CheckError: Invariant failed. Description: celery-k8s execution must be configured in pipeline execution config to launch runs with CeleryK8sRunLauncher
j

johann

08/13/2020, 1:07 PM
In dagit you can click the top left “Instance details”
Can you paste the config block starting with 
run_launcher:
s

szalai1

08/13/2020, 1:08 PM
Copy code
run_launcher:
     module: dagster_celery_k8s
     class: CeleryK8sRunLauncher
     config:
       backend: amqp
broker: <pyamqp://rabbitmquser:rabbitmqpass@dagster-rabbitmq:5672//>
dagster_home:
  env: DAGSTER_HOME
instance_config_map:
  env: DAGSTER_K8S_INSTANCE_CONFIG_MAP
postgres_password_secret:
  env: DAGSTER_K8S_PG_PASSWORD_SECRET
j

johann

08/13/2020, 1:10 PM
Ah that config is coming from the helm deployment
Let me check what the best way to change that is
For now to unblock yourself you could change the config in the helm deployment at 
python_modules/libraries/dagster-k8s/helm/dagster/templates/configmap-instance.yaml
s

szalai1

08/13/2020, 1:16 PM
thank you very much
Powered by