Hey Team, currently i am trying to launch each steps as separate k8 job using k8s_job_executor. And to use k8_job_executor i need to use k8RunLauncher. But i am not able to understand following two things?

run_launcher:
  module: dagster_k8s.launcher
  class: K8sRunLauncher
  config:
    service_account_name: your_service_account
    job_image: my_project/dagster_image:latest
    instance_config_map: dagster-instance
    postgres_password_secret: dagster-postgresql-secret

1. Should I need to create empty config map ? And what's the use of it? 2. Should I need to spin up postgres for postgres_password_secret ? Note : I am using execute_job .

instance = DagsterInstance.get()
result = execute_job(reconstructable(test_job), instance=instance)

Hi Sundara, the

execute_job

api is intended for executing things locally. Once you start deploying to kubernetes, you need things like postgres (for runs to log to) and a docker image with your job code.

@johann I am able to achieve the k8_job_executor with k8RunLauncher using execute_job method by following this chat (https://dagster.slack.com/archives/C014UDS8LAV/p1673505926028249?thread_ts=1673476412.189179&cid=C014UDS8LAV). But still i have following doubts 1. What is the use of the instance config map ? (I can see it has the values similar to dagster.yaml). And why it is compulsory field in k8RunLauncher? 2. And why postgres_password_secret is also compulsory field? 3. Why we should not use execute_job method ? (Any specific constraints)

1. It provides the dagster.yaml to the run and step workers in K8s Jobs. 2. Dagster run and step workers need to be able to write back to a DB. By default we assume postgres. 3. It will be unusual, and some things will be easy to mix up. E.g. if the docker image for the steps is out of date with the code where you called

execute_job

, you get in to undefined behavior and will hit strange errors.

I have some follow up questions. 1. Which means in instance config map, i can upload the dagster.yaml and use that in k8RunLauncher ? (A way to avoid using helm) 2. Is same instance config map, will be shared across multiple repositories ? 3. If i use same image for the steps & the code where i have called execute_job. It should not be a problem right? P.S : No helm install have been used to create dagster-postgres.

1. that should work yeah. 2. Yes it can be shared 3. They’ll need the same code, and access to the same db. If you get that all set up then I think it should work

Okay. This clarifies. Thankyou.