https://dagster.io/ logo
Powered by
Title
c

Charles Leung

12/17/2021, 10:52 PM
Hey all! I'm having an issue deploying on k8s: 
Traceback (most recent call last):
  File "/usr/local/bin/dagit", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/dagit/cli.py", line 239, in main
    cli(auto_envvar_prefix="DAGIT")  # pylint:disable=E1120
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagit/cli.py", line 119, in ui
    **kwargs,
  File "/usr/local/lib/python3.7/site-packages/dagit/cli.py", line 136, in host_dagit_ui
    with get_instance_for_service("dagit") as instance:
  File "/usr/local/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/site-packages/dagster/cli/utils.py", line 12, in get_instance_for_service
    with DagsterInstance.get() as instance:
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 387, in get
    return DagsterInstance.from_config(dagster_home_path)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 402, in from_config
    return DagsterInstance.from_ref(instance_ref)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 423, in from_ref
    run_launcher=instance_ref.run_launcher,
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/ref.py", line 264, in run_launcher
    return self.run_launcher_data.rehydrate() if self.run_launcher_data else None
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/config_class.py", line 85, in rehydrate
    return klass.from_config_value(self, result.value)
  File "/usr/local/lib/python3.7/site-packages/dagster_k8s/launcher.py", line 209, in from_config_value
    return cls(inst_data=inst_data, **config_value)
  File "/usr/local/lib/python3.7/site-packages/dagster_k8s/launcher.py", line 130, in __init__
    kubernetes.config.load_incluster_config()
  File "/usr/local/lib/python3.7/site-packages/kubernetes/config/incluster_config.py", line 121, in load_incluster_config
    try_refresh_token=try_refresh_token).load_and_set(client_configuration)
  File "/usr/local/lib/python3.7/site-packages/kubernetes/config/incluster_config.py", line 54, in load_and_set
    self._load_config()
  File "/usr/local/lib/python3.7/site-packages/kubernetes/config/incluster_config.py", line 73, in _load_config
    raise ConfigException("Service token file does not exist.")
Am I missing a value in values.yaml somewhere?
d

daniel

12/18/2021, 2:12 AM
Hi Charles - this is when using the helm chart?
c

Charles Leung

12/20/2021, 3:31 PM
Hey Daniel! thanks for responding - yes that is when we're using the helm chart.
d

daniel

12/20/2021, 3:34 PM
I see - how did you configure your run launcher in your values.yaml? looks like this is related to the loadInclusterConfig / kubeconfigFile settings
These are the relevant bits from the values.yaml - https://github.com/dagster-io/dagster/blob/master/helm/dagster/values.yaml#L384-L388 . What exactly you put here likely depends on your cluster, but your dagit pod needs to be able to load the correct kubeconfig in order to be able to launch pods for runs
c

Charles Leung

12/20/2021, 3:53 PM
hey Daniel, I just left it as the default k8srunner, trying to load the inclusterconfig is where it fails
d

daniel

12/20/2021, 3:56 PM
and just to double-check, those are logs from your dagit pod I assume?
c

Charles Leung

12/20/2021, 3:57 PM
yes
I am using workload identity for GKE, I am wondering if it has to do with this: https://github.com/kubernetes-client/python-base/issues/159; if this is the case, are there other people trying to use workload identity with dagster?
d

daniel

12/20/2021, 4:00 PM
ah I was just searching around and found a similar error related to GKE Workload Identity (from airflow not dagster, but both systems try to load the cluster config I think): https://stackoverflow.com/questions/57312376/unable-to-execute-airflow-kubernetesexecutor#comment117873811_57398011
I'm looking through https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl now to see if there's some way to generate a kubeconfig file that you could use with the kubeconfigFile setting
c

Charles Leung

12/20/2021, 4:12 PM
hm, I apologize I'm kind of new to k8s. the kubeconfigFile will hold the service token details too?
I appreciate all the help you're giving me 🙂
Hey Daniel! I figured it out actually thanks to the link you gave me.
I created the workload identity and all it took was updating the configuration on the service account
module "k8s-workload-identity" {
  depends_on                      = [kubernetes_namespace.namespace]
  source                          = "terraform-google-modules/kubernetes-engine/google//modules/workload-identity"
  name                            = "${var.namespace_name}-${var.environment}-ksa"
  namespace                       = var.namespace_name
  project_id                      = var.project_id
  automount_service_account_token = true
}
d

daniel

12/20/2021, 6:17 PM
Ahhh, great! We should add this to the docs under deploying on Google Cloud. @Dagster Bot docs document how to run the helm chart when using GKE Workload Identity
d

Dagster Bot

12/20/2021, 6:17 PM
Created issue at: https://github.com/dagster-io/dagster/issues/6021
❤️ 1
2 Views
#dagster-supportJoin Slack
Powered by