https://dagster.io/ logo
#announcements
Title
# announcements
k

Ken

03/20/2020, 12:59 AM
Hello Guys, I am trying to run my pipelines in a Kubernetes Clusters, from my local machine, using the run_launcher "dagster_k8s.launcher" I am seeing this error:
Copy code
dagster.core.errors.DagsterInvalidConfigError: Errors whilst loading configuration for {'service_account_name': <class 'str'>, 'job_image': <class 'str'>, 'in
stance_config_map': <class 'str'>, 'dagster_home': <class 'str'>, 'image_pull_secrets': Field(<dagster.config.config_type.Noneable object at 0x7f1646c88250>,
default=@, is_required=False), 'image_pull_policy': Field(<dagster.config.config_type.String object at 0x7f164f18ce90>, default=Always, is_required=False), 'j
ob_namespace': <class 'str'>, 'env_config_maps': Field(<dagster.config.config_type.Noneable object at 0x7f162684a910>, default=@, is_required=False), 'env_sec
rets': Field(<dagster.config.config_type.Noneable object at 0x7f162684ab10>, default=@, is_required=False)}.
    Error 1: Fields "['kubeconfig_file', 'load_kubeconfig']" are not defined at document config root. Available fields: "['dagster_home', 'env_config_maps', '
env_secrets', 'image_pull_policy', 'image_pull_secrets', 'instance_config_map', 'job_image', 'job_namespace', 'service_account_name']"
To my understanding, shouldn't
kubeconfig_file
and
load_kubeconfig
be accepted in the dagster.yaml ? Details posted in thread of this post.
My Dagster.yaml file is as follow:
Copy code
run_launcher:
  module: dagster_k8s.launcher
  class: K8sRunLauncher
  config:
    service_account_name: "environments/tester-imagery-service-account.json"
    job_image: "<http://gcr.io/tester-gcr-registry/dagster_demo_pipeline_image:v1|gcr.io/tester-gcr-registry/dagster_demo_pipeline_image:v1>"
    instance_config_map: "tester-dagster-demo"
    job_namespace: dagster_demo_pipeline_namespace
    dagster_home: "/tmp"
    load_kubeconfig: True
    kubeconfig_file: "environments/kube.config"
Having that, I ran
dagster pipeline execute -f ./pipelines/main_pipeline.py -n process_image_pipeline -d cloud -p cloud
=> See error.
Reference for where I get the configuration parameters for k8s launcher: ...github/dagster/python_modules/libraries/dagster-k8s/dagster_k8s/launcher.py ( line 57 )
a

alex

03/20/2020, 1:07 AM
the actual config schema is down on line 124
i think if you remove ‘kubeconfig_file’ and ‘load_kubeconfig’ from your yaml it should work
as per the error message piece
Error 1: Fields “[‘kubeconfig_file’, ‘load_kubeconfig’]” are not defined at document config root. Available fields: “[‘dagster_home’, ‘env_config_maps’, '
env_secrets’, ‘image_pull_policy’, ‘image_pull_secrets’, ‘instance_config_map’, ‘job_image’, ‘job_namespace’, ‘service_account_name’]”
it is indeed odd that the config schema is out of sync with the constructor you are looking at cc @nate / @max
k

Ken

03/20/2020, 1:13 AM
Thanks Alex! I did start out with not having those 2 lines. But, I currently want to test run my local pipeline in Kubernetes clusters. And without those 2 lines, the run_launcher seems to think it is being run in a Kubernetes environment, and it tries to seek for Service Host/Port env variables ( which my local environment does not have ) Error message when kube_config is not used:
Copy code
File ".../lib/python3.7/site-packages/kubernetes/config/incluster_config.py", line 51, in _load_config
    raise ConfigException("Service host/port is not set.")
kubernetes.config.config_exception.ConfigException: Service host/port is not set.
a

alex

03/20/2020, 1:17 AM
makes sense, let me throw a diff up to add those two fields to the config
seems like it was just an mistake on our end, apologies for the thrash
k

Ken

03/20/2020, 1:19 AM
That will be awesome No sweat! It happens. I look forward to the update!
a

alex

03/20/2020, 1:22 AM
if you have the repo checked out locally you could
pip install -e python_modules/libraries/dagster-k8s
and make these modifications https://dagster.phacility.com/D2296
👍 1
n

nate

03/20/2020, 1:24 AM
yes, a miss on our part - and thanks Alex for the quick diff
k

Ken

03/20/2020, 1:25 AM
Cool, I will test it out now! I also appreciate the quick diff, thanks Alex! Update: It's working!!
🎉 1
Hello @alex, Thanks again for your code-diff yesterday. After applying that patch, unfortunately, I am seeing another issue: The pipeline will run error-free in the local machine, rather than deploying a job in the Kubernetes Clusters. Context of My Project: I am running a small pipeline. Locally, it works, with an empty dagster.yaml file. The pipeline runs in my local machine. I now want to: • Local Machine: perform
dagster pipeline execute -f ./pipelines/main_pipeline.py -n process_image_pipeline -d cloud -p cloud
• The pipeline will be executed as a Kubernetes Job on Google GKE. Resources: + My Dagster.yaml file:
Copy code
run_launcher:
  module: dagster_k8s.launcher
  class: K8sRunLauncher
  config:
    service_account_name: "environments/tester-imagery-service-account.json"
    job_image: "<http://gcr.io/tester-gcr-registry/dagster_demo_pipeline_image:v1|gcr.io/tester-gcr-registry/dagster_demo_pipeline_image:v1>"
    instance_config_map: "tester-dagster-demo"
    job_namespace: dagster_demo_pipeline_namespace
    dagster_home: "/tmp"
    load_kubeconfig: True
    kubeconfig_file: "environments/kube.config"
telemetry:
  enabled: true
  instance_id: 8a67c01a-34c5-4161-ab60-cc4c1de95a86
+ The documentation that I used: https://github.com/dagster-io/dagster/blob/master/docs/sections/deploying/k8s.rst and source code at "github/dagster/python_modules/libraries/dagster-k8s/dagster_k8s/launcher.py" + Version of Dagster: latest master branch, With patch from https://dagster.phacility.com/D2296 ( addition of the 2 configuration lines for kube_config ) Is there anything I am missing ? or am I misunderstanding anything ?
m

max

03/21/2020, 1:33 AM
Hi @Ken,
dagster pipeline execute
actually bypasses the run launcher. We should add a command line utility that uses the run launcher! Should be pretty straightforward
k

Ken

03/21/2020, 1:36 AM
Thanks for the quick response, Max! I look forward to seeing what you guys come up with.
m

max

03/21/2020, 2:35 AM
k

Ken

03/21/2020, 6:38 AM
Thanks Max, It looks awesome! I do not know how to leave comments in that Web Service, so I hope I could leave it here: 1. >>> File: python_modules/dagster/dagster/cli/pipeline.py Line: 350 New Change: ( added )
Copy code
instructions=get_pipeline_instructions('execute')
Perhaps you meant
Copy code
instructions=get_pipeline_instructions('launch')
Since that will generate a templated docs for the proposed
dagster pipeline launch
command. <<< 2. >>> I propose that we add
Copy code
group.add_command(pipeline_launch_command)
To: File: python_modules/dagster/dagster/cli/pipeline.py Line: 42 So that user can perform
dagster pipeline launch
. At the moment, running that command will throw error
Error: No such command 'launch'.
<<< Other than that, this code revision is now triggering the kubernetes.client package successfully upon
dagster pipeline launch
Again, I really appreciate your work!!
GKE Workload has initiated, using the new launch command!
n

nate

03/21/2020, 3:41 PM
great - thanks for the feedback @Ken I left your notes in the diff for Max—glad this is working!