Hello Guys I am trying to run my pipelines in a Kubernetes C dagster #announcements

Hello Guys, I am trying to run my pipelines in a ...

Ken

03/20/2020, 12:59 AM

Hello Guys, I am trying to run my pipelines in a Kubernetes Clusters, from my local machine, using the run_launcher "dagster_k8s.launcher" I am seeing this error:

Copy code

dagster.core.errors.DagsterInvalidConfigError: Errors whilst loading configuration for {'service_account_name': <class 'str'>, 'job_image': <class 'str'>, 'in
stance_config_map': <class 'str'>, 'dagster_home': <class 'str'>, 'image_pull_secrets': Field(<dagster.config.config_type.Noneable object at 0x7f1646c88250>,
default=@, is_required=False), 'image_pull_policy': Field(<dagster.config.config_type.String object at 0x7f164f18ce90>, default=Always, is_required=False), 'j
ob_namespace': <class 'str'>, 'env_config_maps': Field(<dagster.config.config_type.Noneable object at 0x7f162684a910>, default=@, is_required=False), 'env_sec
rets': Field(<dagster.config.config_type.Noneable object at 0x7f162684ab10>, default=@, is_required=False)}.
    Error 1: Fields "['kubeconfig_file', 'load_kubeconfig']" are not defined at document config root. Available fields: "['dagster_home', 'env_config_maps', '
env_secrets', 'image_pull_policy', 'image_pull_secrets', 'instance_config_map', 'job_image', 'job_namespace', 'service_account_name']"

To my understanding, shouldn't

kubeconfig_file

and

load_kubeconfig

be accepted in the dagster.yaml ? Details posted in thread of this post.

Ken

03/20/2020, 12:59 AM

My Dagster.yaml file is as follow:

Copy code

run_launcher:
  module: dagster_k8s.launcher
  class: K8sRunLauncher
  config:
    service_account_name: "environments/tester-imagery-service-account.json"
    job_image: "<http://gcr.io/tester-gcr-registry/dagster_demo_pipeline_image:v1|gcr.io/tester-gcr-registry/dagster_demo_pipeline_image:v1>"
    instance_config_map: "tester-dagster-demo"
    job_namespace: dagster_demo_pipeline_namespace
    dagster_home: "/tmp"
    load_kubeconfig: True
    kubeconfig_file: "environments/kube.config"

Having that, I ran

dagster pipeline execute -f ./pipelines/main_pipeline.py -n process_image_pipeline -d cloud -p cloud

=> See error.

Ken

03/20/2020, 1:04 AM

Reference for where I get the configuration parameters for k8s launcher: ...github/dagster/python_modules/libraries/dagster-k8s/dagster_k8s/launcher.py ( line 57 )

alex

03/20/2020, 1:07 AM

the actual config schema is down on line 124

alex

03/20/2020, 1:08 AM

i think if you remove ‘kubeconfig_file’ and ‘load_kubeconfig’ from your yaml it should work

alex

03/20/2020, 1:08 AM

as per the error message piece

Error 1: Fields “[‘kubeconfig_file’, ‘load_kubeconfig’]” are not defined at document config root. Available fields: “[‘dagster_home’, ‘env_config_maps’, '

env_secrets’, ‘image_pull_policy’, ‘image_pull_secrets’, ‘instance_config_map’, ‘job_image’, ‘job_namespace’, ‘service_account_name’]”

alex

03/20/2020, 1:10 AM

it is indeed odd that the config schema is out of sync with the constructor you are looking at cc @nate / @max

Ken

03/20/2020, 1:13 AM

Thanks Alex! I did start out with not having those 2 lines. But, I currently want to test run my local pipeline in Kubernetes clusters. And without those 2 lines, the run_launcher seems to think it is being run in a Kubernetes environment, and it tries to seek for Service Host/Port env variables ( which my local environment does not have ) Error message when kube_config is not used:

Copy code

File ".../lib/python3.7/site-packages/kubernetes/config/incluster_config.py", line 51, in _load_config
    raise ConfigException("Service host/port is not set.")
kubernetes.config.config_exception.ConfigException: Service host/port is not set.

alex

03/20/2020, 1:17 AM

makes sense, let me throw a diff up to add those two fields to the config

alex

03/20/2020, 1:17 AM

seems like it was just an mistake on our end, apologies for the thrash

Ken

03/20/2020, 1:19 AM

That will be awesome No sweat! It happens. I look forward to the update!

alex

03/20/2020, 1:22 AM

if you have the repo checked out locally you could

pip install -e python_modules/libraries/dagster-k8s

and make these modifications https://dagster.phacility.com/D2296

👍 1

nate

03/20/2020, 1:24 AM

yes, a miss on our part - and thanks Alex for the quick diff

Ken

03/20/2020, 1:25 AM

Cool, I will test it out now! I also appreciate the quick diff, thanks Alex! Update: It's working!!

🎉 1

Ken

03/21/2020, 1:05 AM

Hello @alex, Thanks again for your code-diff yesterday. After applying that patch, unfortunately, I am seeing another issue: The pipeline will run error-free in the local machine, rather than deploying a job in the Kubernetes Clusters. Context of My Project: I am running a small pipeline. Locally, it works, with an empty dagster.yaml file. The pipeline runs in my local machine. I now want to: • Local Machine: perform

dagster pipeline execute -f ./pipelines/main_pipeline.py -n process_image_pipeline -d cloud -p cloud

• The pipeline will be executed as a Kubernetes Job on Google GKE. Resources: + My Dagster.yaml file:

Copy code

run_launcher:
  module: dagster_k8s.launcher
  class: K8sRunLauncher
  config:
    service_account_name: "environments/tester-imagery-service-account.json"
    job_image: "<http://gcr.io/tester-gcr-registry/dagster_demo_pipeline_image:v1|gcr.io/tester-gcr-registry/dagster_demo_pipeline_image:v1>"
    instance_config_map: "tester-dagster-demo"
    job_namespace: dagster_demo_pipeline_namespace
    dagster_home: "/tmp"
    load_kubeconfig: True
    kubeconfig_file: "environments/kube.config"
telemetry:
  enabled: true
  instance_id: 8a67c01a-34c5-4161-ab60-cc4c1de95a86

+ The documentation that I used: https://github.com/dagster-io/dagster/blob/master/docs/sections/deploying/k8s.rst and source code at "github/dagster/python_modules/libraries/dagster-k8s/dagster_k8s/launcher.py" + Version of Dagster: latest master branch, With patch from https://dagster.phacility.com/D2296 ( addition of the 2 configuration lines for kube_config ) Is there anything I am missing ? or am I misunderstanding anything ?

max

03/21/2020, 1:33 AM

Hi @Ken,

dagster pipeline execute

actually bypasses the run launcher. We should add a command line utility that uses the run launcher! Should be pretty straightforward

Ken

03/21/2020, 1:36 AM

Thanks for the quick response, Max! I look forward to seeing what you guys come up with.

max

03/21/2020, 2:35 AM

ok, sketch is up at https://dagster.phacility.com/D2303

Ken

03/21/2020, 6:38 AM

Thanks Max, It looks awesome! I do not know how to leave comments in that Web Service, so I hope I could leave it here: 1. >>> File: python_modules/dagster/dagster/cli/pipeline.py Line: 350 New Change: ( added )

Copy code

instructions=get_pipeline_instructions('execute')

Perhaps you meant

Copy code

instructions=get_pipeline_instructions('launch')

Since that will generate a templated docs for the proposed

dagster pipeline launch

command. <<< 2. >>> I propose that we add

Copy code

group.add_command(pipeline_launch_command)

To: File: python_modules/dagster/dagster/cli/pipeline.py Line: 42 So that user can perform

dagster pipeline launch

. At the moment, running that command will throw error

Error: No such command 'launch'.

<<< Other than that, this code revision is now triggering the kubernetes.client package successfully upon

dagster pipeline launch

Again, I really appreciate your work!!

Ken

03/21/2020, 7:55 AM

GKE Workload has initiated, using the new launch command!

nate

03/21/2020, 3:41 PM

great - thanks for the feedback @Ken I left your notes in the diff for Max—glad this is working!

Open in Slack

Previous Next