Hello wave I am trying to run a local Dagster with a `k8sRun

Question

Hello wave I am trying to run a local Dagster with a `k8sRunLauncher` smile My goal is to host dagit locally and run job s steps on the EKS cluster Do you know if this is supported by Dagster I currently have the following configuration ```run launcher module dagster k8s launcher class K8sRunLauncher config dagster home usr src app job image lt my code image gt 0 0 1 image pull policy Always instance config map dagster instance job namespace system dagster load incluster config false kubeconfig file Users alexandreguitton kube config postgres password secret dagster postgresql volumes configMap defaultMode 420 name dagster instance name dagster instance resources limits cpu 1 memory 200Mi requests cpu 1 memory 200Mi service account name dagster``` It runs the pod correctly but the pod can t connect because of `load incluster config` set to false as the configuration is passed in the command and it doesn t seem that `instance config map` works or maybe I didn t understand how it works confused Any help would be greatly appreciated smile

Alexandre Guitton · Answer

Up anyone to help me on this confused

daniel · Answer

Hey Alexandre it might take some work on our side to fully support this although i think its a legitimate use case Can you share the exact error that you re seeing when you try to do this

Alexandre Guitton · Answer

Hey Thanks for the reply slightly smiling face So my current local configuration of `dagster yaml` is ```run launcher module dagster k8s launcher class K8sRunLauncher config dagster home usr src app job image env DAGSTER JOB IMAGE image pull policy Always instance config map dagster instance job namespace system dagster load incluster config false kubeconfig file ~ kube config postgres password secret dagster postgresql volumes configMap defaultMode 420 name dagster instance name dagster instance resources requests cpu 1 memory 200Mi service account name dagster env vars DAGSTER PG HOSTNAME DAGSTER PG DB NAME DAGSTER PG PORT DAGSTER PG USERNAME TMP DAGSTER BASE DIR DAGSTER JOB IMAGE``` When I launch the job it has the following pod definition ```apiVersion v1 kind Pod metadata annotations eks privileged creationTimestamp 2023 02 02T16 52 46Z generateName dagster run 05ac286c f6fc 4bb2 a7a7 d73d0daa6fa9 labels run worker dagster dagster dagster 1 1 14 controller uid 70971e8f f283 4cee bf0a f8c9645ba518 dagster job ASSET JOB dagster run id 05ac286c f6fc 4bb2 a7a7 d73d0daa6fa9 job name dagster run 05ac286c f6fc 4bb2 a7a7 d73d0daa6fa9 name dagster run 05ac286c f6fc 4bb2 a7a7 d73d0daa6fa9 vktln namespace system dagster ownerReferences apiVersion batch v1 blockOwnerDeletion true controller true kind Job name dagster run 05ac286c f6fc 4bb2 a7a7 d73d0daa6fa9 uid 70971e8f f283 4cee bf0a f8c9645ba518 resourceVersion 7411517 uid 245a9713 bde9 4962 bd92 20c6f8e9b814 spec containers args dagster api execute run class ExecuteRunArgs instance ref class InstanceRef compute logs data class ConfigurableClassData class name LocalComputeLogManager config yaml base dir n env TMP DAGSTER BASE DIR n module name dagster core storage local compute log manager custom instance class data null event storage data class ConfigurableClassData class name PostgresEventLogStorage config yaml postgres db n db name dagster n hostname n env DAGSTER PG HOSTNAME n params n password n env DAGSTER PG PASSWORD n port n env DAGSTER PG PORT n username n env DAGSTER PG USERNAME n module name dagster postgres local artifact storage data class ConfigurableClassData class name LocalArtifactStorage config yaml base dir n env TMP DAGSTER BASE DIR n module name dagster core storage root run coordinator data class ConfigurableClassData class name DefaultRunCoordinator config yaml n module name dagster core run coordinator run launcher data class ConfigurableClassData class name K8sRunLauncher config yaml dagster home usr src app nenv vars n DAGSTER PG HOSTNAME n DAGSTER PG DB NAME n DAGSTER PG PORT n DAGSTER PG USERNAME n TMP DAGSTER BASE DIR n DAGSTER JOB IMAGE nimage pull policy Always ninstance config map dagster instance njob image n env DAGSTER JOB IMAGE njob namespace system dagster nkubeconfig file ~ kube config nload incluster config false npostgres password secret dagster postgresql nresources n requests n cpu 1 n memory 200Mi nservice account name dagster nvolumes n configMap n defaultMode 420 n name dagster instance n name dagster instance n module name dagster k8s launcher run storage data class ConfigurableClassData class name PostgresRunStorage config yaml postgres db n db name dagster n hostname n env DAGSTER PG HOSTNAME n params n password n env DAGSTER PG PASSWORD n port n env DAGSTER PG PORT n username n env DAGSTER PG USERNAME n module name dagster postgres schedule storage data class ConfigurableClassData class name PostgresScheduleStorage config yaml postgres db n db name dagster n hostname n env DAGSTER PG HOSTNAME n params n password n env DAGSTER PG PASSWORD n port n env DAGSTER PG PORT n username n env DAGSTER PG USERNAME n module name dagster postgres scheduler data class ConfigurableClassData class name DagsterDaemonScheduler config yaml n module name dagster core scheduler secrets loader data null settings storage data class ConfigurableClassData class name DagsterPostgresStorage config yaml postgres db n db name dagster n hostname n env DAGSTER PG HOSTNAME n params n password n env DAGSTER PG PASSWORD n port n env DAGSTER PG PORT n username n env DAGSTER PG USERNAME n module name dagster postgres pipeline origin class PipelinePythonOrigin pipeline name ASSET JOB repository origin class RepositoryPythonOrigin code pointer class ModuleCodePointer fn name defs module github slack working directory usr src app container context container image null entry point dagster executable path root cache pypoetry virtualenvs github slack VA82Wl8V py3 9 bin python pipeline run id 05ac286c f6fc 4bb2 a7a7 d73d0daa6fa9 set exit code on failure null env name DAGSTER RUN JOB NAME value ASSET JOB name DAGSTER HOME value usr src app name DAGSTER PG PASSWORD valueFrom secretKeyRef key postgresql password name dagster postgresql name DAGSTER PG PORT value 5432 name TMP DAGSTER BASE DIR value usr src app tmp name DAGSTER JOB IMAGE value name DAGSTER PG DB NAME value dagster name DAGSTER PG USERNAME value dagster name DAGSTER PG HOSTNAME value image imagePullPolicy Always name dagster resources requests cpu 1 memory 200Mi terminationMessagePath dev termination log terminationMessagePolicy File volumeMounts mountPath var run secrets kubernetes io serviceaccount name kube api access 7x5rc readOnly true dnsPolicy ClusterFirst enableServiceLinks true nodeName ip 10 200 17 172 eu central 1 compute internal preemptionPolicy PreemptLowerPriority priority 0 restartPolicy Never schedulerName default scheduler securityContext serviceAccount dagster serviceAccountName dagster terminationGracePeriodSeconds 30 tolerations effect NoExecute key operator Exists tolerationSeconds 300 effect NoExecute key operator Exists tolerationSeconds 300 volumes configMap defaultMode 420 name dagster instance name dagster instance name kube api access 7x5rc projected defaultMode 420 sources serviceAccountToken expirationSeconds 3607 path token configMap items key ca crt path ca crt name kube root ca crt downwardAPI items fieldRef apiVersion v1 fieldPath metadata namespace path namespace``` And the pod fails with the following error ``` class DagsterEvent event specific data class EngineEventData error class SerializableErrorInfo cause null cls name ConfigException context null message kubernetes config config exception ConfigException Invalid kube config file No configuration found n stack File usr local lib python3 9 site packages dagster grpc impl py line 130 in core execute run n yield from execute run iterator n File usr local lib python3 9 site packages dagster core execution api py line 1097 in iter n yield from self execution context manager prepare context n File usr local lib python3 9 site packages dagster utils init py line 521 in generate setup events n obj = next self generator n File usr local lib python3 9 site packages dagster core execution context creation pipeline py line 333 in orchestration context event generator n executor = create executor context creation data n File usr local lib python3 9 site packages dagster core execution context creation pipeline py line 426 in create executor n return creation fn init context n File usr local lib python3 9 site packages dagster k8s executor py line 96 in k8s job executor n run launcher = init context instance run launcher n File usr local lib python3 9 site packages dagster core instance init py line 659 in run launcher n launcher = cast InstanceRef self ref run launcher n File usr local lib python3 9 site packages dagster core instance ref py line 491 in run launcher n return self run launcher data rehydrate if self run launcher data else None n File usr local lib python3 9 site packages dagster serdes config class py line 101 in rehydrate n return klass from config value self check not none result value n File usr local lib python3 9 site packages dagster k8s launcher py line 200 in from config value n return cls inst data=inst data config value n File usr local lib python3 9 site packages dagster k8s launcher py line 87 in init n kubernetes config load kube config kubeconfig file n File usr local lib python3 9 site packages kubernetes config kube config py line 813 in load kube config n loader = get kube config loader n File usr local lib python3 9 site packages kubernetes config kube config py line 770 in get kube config loader n raise ConfigException n marker end null marker start null metadata entries event type value ENGINE EVENT logging tags message An exception was thrown during execution that is likely a framework error rather than an error in user code pid null pipeline name ASSET JOB solid handle null step handle null step key null step kind value null class DagsterEvent event specific data null event type value PIPELINE FAILURE logging tags message This run has been marked as failed from outside the execution context pid null pipeline name ASSET JOB solid handle null step handle null step key null step kind value null Stream closed EOF for system dagster dagster run 05ac286c f6fc 4bb2 a7a7 d73d0daa6fa9 vktln dagster ``` I guess it fails because the `run launch` configuration is passed through the command line ```job namespace system dagster nkubeconfig file ~ kube config nload incluster config false``` But I expected that `instance config map` to allow the launched job to use the `dagster instance` configmap instead which has the following definition

Alexandre Guitton · Answer

```apiVersion v1 data dagster yaml scheduler n module dagster core scheduler n class DagsterDaemonScheduler n nschedule storage n module dagster postgres schedule storage n class PostgresScheduleStorage n config n postgres db n username dagster n password n env DAGSTER PG PASSWORD n hostname n db name dagster n port 5432 n params n n nrun launcher n module dagster k8s n class K8sRunLauncher n config n load incluster config true n job namespace system dagster n image pull policy Always n service account name dagster n dagster home opt dagster dagster home n instance config map dagster instance n postgres password secret dagster postgresql n nrun storage n module dagster postgres run storage n class PostgresRunStorage n config n postgres db n username dagster n password n env DAGSTER PG PASSWORD n hostname n db name dagster n port 5432 n params n n nevent log storage n module dagster postgres event log n class PostgresEventLogStorage n config n postgres db n username dagster n password n env DAGSTER PG PASSWORD n hostname n db name dagster n port 5432 n params n ncompute logs n module dagster core storage noop compute log manager n class NoOpComputeLogManager n ntelemetry n enabled true n kind ConfigMap metadata annotations dagster system dagster creationTimestamp 2023 01 30T17 03 33Z labels app dagster Helm chart dagster 1 1 14 heritage Helm release dagster name dagster instance namespace system dagster resourceVersion 6397349 uid 1cb5fe9b ea71 4ee5 98db 5e6940a128d7```

daniel · Answer

Are you using the k8s job executor as well

daniel · Answer

I would suggest starting with getting it working without that that executor requires the pod that it spins up to also be able to launch pods for each step so its probably a bit tricky to get both working with the same config

Alexandre Guitton · Answer

Yes my Definitions is ```defs = Definitions assets= airbyte assets hex notebook asset + dbt assets assets= airbyte assets + dbt assets resources= dbt dbt cli resource configured project dir DBT PROJECT DIR profiles dir DBT PROJECT DIR hex hex resource configured api key env HEX TOKEN jobs= update dash job executor=k8s job executor schedules= update dash schedule ``` If I deploy to Kubernetes my user deployment it works perfectly but it means that every time I want to test my code e g to test Kubernetes Service Account accesses I have to push to my k8s cluster the user deployment and run to test I was wondering if I could avoid the user deployment on k8s by just running dagit locally with the job execution on the k8s cluster

daniel · Answer

does it work if you use the default executor instead commenting out the executor= part

daniel · Answer

just to start

daniel · Answer

if it does then we can move on to getting it working with the executor

Alexandre Guitton · Answer

Yes it seems to work fine slightly smiling face

daniel · Answer

OK I think to make this work we would need to expose a load incluster config config field on the executor and you would need to set it to True so that it loads the k8s config from within the cluster instead of trying to find your local kubeconfig file

daniel · Answer

alternately you could try to include the kubeconfig that your local machine is using on the launched pod for the dagster job i m not actually 100 sure if that will work

daniel · Answer

is this something you d be willing to file a feature request issue for on github

daniel · Answer

I m assuming its important to you to use the k8s job executor i e you want each step to run in its own pod

Alexandre Guitton · Answer

Yes we really need it But I think I am missing something to understand the solution Why does the run pod try to load the kubeconfig file when I run from a local and not when it s from a Kubernetes user deployment Is this because it defaults to the K8sRunLauncher configuration If so in that case what is the `instance config map` field for

daniel · Answer

the run pod tries to load the kubeconfig file so that it can spin up a pod for each step

daniel · Answer

I think instance config map is more or less deprecated

daniel · Answer

although I see how it would be useful in this specific situation

daniel · Answer

instead the instance config to use is passed in as an argument to the command

Alexandre Guitton · Answer

Just created the <https github com dagster io dagster issues 12049|feature request> Thank you for this quick and efficient discussion dagster bot responded by community

TadasG · Answer

maybe you guys managed to fix this I m wondering how are you able to join local dagit vs kubernetes dagit I m getting ```dagster check CheckError Expected non None value Pipeline run with id 82c96833 8188 4cbd bcaa c382593a4158 not found for run execution ``` Dagit on K8s cannot find this job and fails since the executor is running on my machine gt gt

daniel · Answer

They ll need to share storage for this to work for example in an external postgres DB that both local and Kubernetes have access to

TadasG · Answer

works thanks slightly smiling face