The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

hi,
is there any way to run pipelines in a single `k8s` job using `celery_k8s_job_executor`?
starting a job per solid, in case of small/fast solids makes pipelines slower.

Take a look at `python_modules/libraries/dagster-k8s/dagster_k8s/launcher.py`

Currently we only use celery to do each execution step in separate pods, does your use case require celery?

I think no. based on this : <https://docs.dagster.io/overview/run-launchers-executors/executor> I thought there is no way to run k8s jobs without celery

Gotcha. We have 2 concepts for deployement, run launchers and executors. Run launchers prepare the execution of the entire pipeline, while the executor is what is responsible for running the individual solids

In your case what I think you want is the k8s run launcher (which will kick off a pipeline run in a k8s job), then probably the multiprocess executor (which will run the solids locally in that pod)

ahh that makes sense, I felt that I'm missing some concept, thank you.

You’re welcome! We’re working on making it more clear in the docs, thanks for working through it

I’ll also add you to <#C014N0PK37E|k8s-deployment>

hmm, I've removed the `celery_k8s_job_executor` from my mode and set `multiproc`. but now getting this error:
`dagster.check.CheckError: Invariant failed. Description: celery-k8s execution must be configured in pipeline execution config to launch runs with CeleryK8sRunLauncher`

In dagit you can click the top left “Instance details”

Can you paste the config block starting with `run_launcher:`

```run_launcher:
     module: dagster_celery_k8s
     class: CeleryK8sRunLauncher
     config:
       backend: amqp
broker: <pyamqp://rabbitmquser:rabbitmqpass@dagster-rabbitmq:5672//>
dagster_home:
  env: DAGSTER_HOME
instance_config_map:
  env: DAGSTER_K8S_INSTANCE_CONFIG_MAP
postgres_password_secret:
  env: DAGSTER_K8S_PG_PASSWORD_SECRET```

Ah that config is coming from the helm deployment

Let me check what the best way to change that is

For now to unblock yourself you could change the config in the helm deployment at `python_modules/libraries/dagster-k8s/helm/dagster/templates/configmap-instance.yaml`

Replace the run launcher config with <https://github.com/dagster-io/dagster/blob/24a576bd26fe779af907dba01916986cbda9cbd8/python_modules/libraries/dagster-k8s/dagster_k8s/launcher.py#L43-L50>