https://dagster.io/ logo
Powered by
#deployment-kubernetes
Title
# deployment-kubernetes
k

Kevin

06/22/2020, 6:11 PM
Hi everyone 🙂 I'm trying to setup a vanilla dagster deployment in k8s, but i've come across an error whenever I try to execute a pipeline :s I've specified in the dagster.yaml: ... run_launcher: module: dagster_k8s.launcher class: CeleryK8sRunLauncher ...
m

max

06/22/2020, 7:11 PM
@nate
j

johann

06/22/2020, 7:46 PM
If you're deploying with the helm chart, it's currently set to celery-k8s https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-k8s/helm/dagster/templates/configmap-instance.yaml#L30 misread your message, sorry
This should be fixed with
Copy code
execution:
  celery-k8s:
    # ... configs here
in your pipeline config
k

Kevin

06/22/2020, 9:11 PM
hi @johann 🙂 the example was helpful, as it makes sense the the pipelines should be defined to use celery. But I feel I am still missing some config somewhere as I am still getting the same error
j

johann

06/22/2020, 9:34 PM
@Kevin Could you post your pipeline config?
k

Kevin

06/22/2020, 9:41 PM
j

johann

06/22/2020, 9:47 PM
@Kevin It looks like
Copy code
execution:
  celery-k8s:
isn't defined
Copy code
execution:
  celery-k8s:
    config:
      job_image: '<http://my_repo.com/image_name:latest|my_repo.com/image_name:latest>'
      job_namespace: 'some-namespace'
      broker: '<pyamqp://guest@localhost//>'  # Optional[str]: The URL of the Celery broker
      backend: 'rpc://' # Optional[str]: The URL of the Celery results backend
      include: ['my_module'] # Optional[List[str]]: Modules every worker should import
      config_source: # Dict[str, Any]: Any additional parameters to pass to the
          #...       # Celery workers. This dict will be passed as the `config_source`
          #...       # argument of celery.Celery().
k

Kevin

06/22/2020, 9:50 PM
in a k8s deployment how would you define the optional parameters? as these pods, i imagine, would be spun up and down based on need
j

johann

06/22/2020, 9:55 PM
If I'm understanding your question correctly, the urls wouldn't need point to ephemeral pods- the broker would point to a deployment of redis, rabbitmq, etc.
Same with the result backend
k

Kevin

06/22/2020, 10:00 PM
I'm still getting the same error despite adding the:
Copy code
execution:
  celery-k8s:
    config:
      job_image: '<http://my_repo.com/image_name:latest|my_repo.com/image_name:latest>'
      job_namespace: 'some-namespace'
referring to a dagster-celery-worker image with the following dockerfile:
Copy code
FROM python:3.7.7

# ADD build_cache/ /

RUN apt-get update -yqq && \
    apt-get install -yqq cron && \
    pip install \
     dagster \
     dagster-graphql \
     dagster-postgres \
     dagster-cron \
     dagster-celery[flower,redis,kubernetes] \
     dagster-k8s
j

johann

06/22/2020, 10:46 PM
In that case I’m not sure, @nate
k

Kevin

06/22/2020, 11:09 PM
well thank you @johann 🙂 you've at least pointed me in the right direction!
n

nate

06/22/2020, 11:12 PM
@Kevin did you add that 
execution:
YAML snippet to your pipeline run config in the playground?
it should go there, not in 
dagster.yaml
- we split up configuration between 
dagster.yaml
for instance-wide defaults, and in the pipeline run config (in the playground) for pipeline-specific configuration
k

Kevin

06/22/2020, 11:15 PM
😮 I added it to the dagster.yaml, ok but if the 'standard' k8s deployment is to use celery why not define it in the dagster.yaml?
Thank you @nate the pipeline got started execution properly now 🙂 but is there not a way to globally define this for all pipelines to be executed?
n

nate

06/22/2020, 11:23 PM
awesome—thanks for slogging through this, we haven’t gotten around to documenting all the new stuff yet! you can configure broker/backend globally on the run launcher, but our assumption is that job_image is often going to be different for different pipelines, so it’s set only per run right now
but, all this is still brand new and in flux, so stay tuned 🙂
k

Kevin

06/23/2020, 5:29 PM
Hi @nate! Thank you for your help again! I checked again this morning and I noticed that the execution of the pipeline now just hangs on "starting"... I'm using the docker image i posted above but with the addition of the copied pipeline files :s
Copy code
Traceback (most recent call last):
  File "/usr/local/bin/dagster-graphql", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/cli.py", line 219, in main
    cli(obj={})  # pylint:disable=E1120
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/cli.py", line 211, in ui
    execute_query_from_cli(workspace, query, variables, output)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/cli.py", line 96, in execute_query_from_cli
    workspace, query, variables=seven.json.loads(variables) if variables else None
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/cli.py", line 47, in execute_query
    else DagsterInstance.get()
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 239, in get
    return DagsterInstance.from_config(_dagster_home())
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 261, in from_config
    return DagsterInstance.from_ref(instance_ref)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 274, in from_ref
    run_launcher=instance_ref.run_launcher,
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/ref.py", line 205, in run_launcher
    return self.run_launcher_data.rehydrate() if self.run_launcher_data else None
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 351, in rehydrate
    config_dict,
dagster.core.errors.DagsterInvalidConfigError: Errors whilst loading configuration for {'instance_config_map': Field(<dagster.config.source.StringSourceType object at 0x7f4a46f5d1d0>, default=@, is_required=True), 'postgres_password_secret': Field(<dagster.config.source.StringSourceType object at 0x7f4a46f5d1d0>, default=@, is_required=True), 'dagster_home': Field(<dagster.config.source.StringSourceType object at 0x7f4a46f5d1d0>, default=/opt/dagster/dagster_home, is_required=False), 'load_incluster_config': Field(<dagster.config.config_type.Bool object at 0x7f4a48e31950>, default=True, is_required=False), 'kubeconfig_file': Field(<dagster.config.config_type.Noneable object at 0x7f4a2a2d2c50>, default=None, is_required=False), 'broker': Field(<dagster.config.config_type.Noneable object at 0x7f4a45101ad0>, default=@, is_required=False), 'backend': Field(<dagster.config.config_type.Noneable object at 0x7f4a45101b50>, default=rpc://, is_required=False), 'include': Field(<dagster.config.config_type.Array object at 0x7f4a450a3190>, default=@, is_required=False), 'config_source': Field(<dagster.config.config_type.Noneable object at 0x7f4a450a3950>, default=@, is_required=False), 'retries': Field(<dagster.config.field_utils.Selector object at 0x7f4a48dd58d0>, default={'enabled': {}}, is_required=False)}.
    Error 1: Post processing at path root:instance_config_map of original value {'env': 'DAGSTER_K8S_INSTANCE_CONFIG_MAP'} failed:
(PostProcessingError) - dagster.config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_K8S_INSTANCE_CONFIG_MAP" which is not set. In order for this execution to succeed it must be set in this environment.

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/config/post_process.py", line 72, in _post_process
    new_value = context.config_type.post_process(config_value)
  File "/usr/local/lib/python3.7/site-packages/dagster/config/source.py", line 42, in post_process
    return str(_ensure_env_variable(cfg))
  File "/usr/local/lib/python3.7/site-packages/dagster/config/source.py", line 23, in _ensure_env_variable
    ).format(var=var)

    Error 2: Post processing at path root:postgres_password_secret of original value {'env': 'DAGSTER_K8S_PG_PASSWORD_SECRET'} failed:
(PostProcessingError) - dagster.config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_K8S_PG_PASSWORD_SECRET" which is not set. In order for this execution to succeed it must be set in this environment.

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/config/post_process.py", line 72, in _post_process
    new_value = context.config_type.post_process(config_value)
  File "/usr/local/lib/python3.7/site-packages/dagster/config/source.py", line 42, in post_process
    return str(_ensure_env_variable(cfg))
  File "/usr/local/lib/python3.7/site-packages/dagster/config/source.py", line 23, in _ensure_env_variable
    ).format(var=var)
the error i get from kubectl logs for the run job is:
j

johann

06/23/2020, 5:53 PM
Could you send your 
values.yaml
helm file?
k

Kevin

06/23/2020, 6:04 PM
My pleasure!
Copy code
dagit:
  replicaCount: 1
  # REQUIRED: Dagit image repository and tag to deploy
  image:
    repository: ".../dagster-dagit"
    tag: "v1"
    pullPolicy: Always

pipeline_run:
  # REQUIRED: The Dagster K8s run launchers will invoke job executions in containers from this image
  image:
    repository: ".../dagster-core"
    tag: "v1"
    # Change with caution! If you're using a fixed tag for pipeline run images, changing the image
    # pull policy to anything other than "Always" will use a cached/stale image, which is almost
    # certainly not what you want.
    pullPolicy: Always

celery:
  image:
    repository: ".../dagster-core"
    tag: "v1"
    pullPolicy: Always
  enabled: true

flower:
  enabled: false

ingress:
  enabled: true
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: "nginx"
    <http://nginx.ingress.kubernetes.io/enable-cors|nginx.ingress.kubernetes.io/enable-cors>: "true"
    <http://nginx.ingress.kubernetes.io/affinity|nginx.ingress.kubernetes.io/affinity>: "cookie"
  dagit:
    host: dagit.testing.testing
    path: "/"
    annotations:
      <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: "nginx"
      <http://nginx.ingress.kubernetes.io/enable-cors|nginx.ingress.kubernetes.io/enable-cors>: "true"
      <http://nginx.ingress.kubernetes.io/affinity|nginx.ingress.kubernetes.io/affinity>: "cookie"
  flower:
    host: flower.testing.testing
j

johann

06/23/2020, 6:06 PM
It looks like you're missing postgresql config, which is required for k8s deployment
The defaults are
Copy code
####################################################################################################
# PostgreSQL: Configuration values for postgresql
#
# <https://github.com/kubernetes/charts/blob/master/stable/postgresql/README.md>
#
# A PostgreSQL database is required to run Dagster on Kubernetes. If postgresql.enabled is marked as
# false, the PG credentials specified here will still be used, and should point to an external PG
# database that is accessible from this chart.
####################################################################################################
postgresql:
  # Used by init container to check that db is running. (Even if enabled:false)
  image:
    repository: "postgres"
    tag: "9.6.16"
    pullPolicy: IfNotPresent
  # set postgresql.enabled to be false to disable deploy of a PostgreSQL database and use an
  # existing external PostgreSQL database
  enabled: true
  # set this PostgreSQL hostname when using an external PostgreSQL database
  postgresqlHost: ""
  postgresqlUsername: test
  # Note when changing this password (e.g. in test) that credentials will
  # persist as long as the PVCs do -- see:
  # <https://github.com/helm/charts/issues/12836#issuecomment-524552358>
  postgresqlPassword: test
  postgresqlDatabase: test
  service:
    port: 5432
k

Kevin

06/23/2020, 6:10 PM
Sorry @johann so we are using the values.yaml i sent you as an input to the values.yaml you have from your repo at dagster/python_modules/libraries/dagster-k8s/helm/dagster/ Go to file
within our k8s the helm install does actually instantiate a postgresql and a few celery-worker nodes as well as the dagit image of course
j

johann

06/23/2020, 6:11 PM
Got it
k

Kevin

06/23/2020, 6:12 PM
yeah, trying to keep it as vanilla as possible!
j

johann

06/23/2020, 6:30 PM
Just to confirm the env vars are missing you could ssh into the pod
k

Kevin

06/23/2020, 6:31 PM
the dagit pod or the runjob pod?
j

johann

06/23/2020, 6:32 PM
I believe it should be in either
k

Kevin

06/23/2020, 6:35 PM
these were all found within the dagit image:
Copy code
$DAGSTER_DAGIT_PORT
$DAGSTER_DAGIT_PORT_80_TCP
$DAGSTER_DAGIT_PORT_80_TCP_ADDR
$DAGSTER_DAGIT_PORT_80_TCP_PORT
$DAGSTER_DAGIT_PORT_80_TCP_PROTO
$DAGSTER_DAGIT_SERVICE_HOST
$DAGSTER_DAGIT_SERVICE_PORT
$DAGSTER_DAGIT_SERVICE_PORT_HTTP
$DAGSTER_HOME
$DAGSTER_K8S_CELERY_BACKEND
$DAGSTER_K8S_CELERY_BROKER
$DAGSTER_K8S_INSTANCE_CONFIG_MAP
$DAGSTER_K8S_PG_PASSWORD_SECRET
$DAGSTER_K8S_PIPELINE_RUN_ENV_CONFIGMAP
$DAGSTER_K8S_PIPELINE_RUN_IMAGE
$DAGSTER_K8S_PIPELINE_RUN_IMAGE_PULL_POLICY
$DAGSTER_K8S_PIPELINE_RUN_NAMESPACE
$DAGSTER_PG_PASSWORD
$DAGSTER_POSTGRESQL_PORT
$DAGSTER_POSTGRESQL_PORT_5432_TCP
$DAGSTER_POSTGRESQL_PORT_5432_TCP_ADDR
$DAGSTER_POSTGRESQL_PORT_5432_TCP_PORT
$DAGSTER_POSTGRESQL_PORT_5432_TCP_PROTO
$DAGSTER_POSTGRESQL_SERVICE_HOST
$DAGSTER_POSTGRESQL_SERVICE_PORT
$DAGSTER_POSTGRESQL_SERVICE_PORT_TCP_POSTGRESQL
$DAGSTER_RABBITMQ_PORT
$DAGSTER_RABBITMQ_PORT_15672_TCP
$DAGSTER_RABBITMQ_PORT_15672_TCP_ADDR
$DAGSTER_RABBITMQ_PORT_15672_TCP_PORT
$DAGSTER_RABBITMQ_PORT_15672_TCP_PROTO
$DAGSTER_RABBITMQ_PORT_25672_TCP
$DAGSTER_RABBITMQ_PORT_25672_TCP_ADDR
$DAGSTER_RABBITMQ_PORT_25672_TCP_PORT
$DAGSTER_RABBITMQ_PORT_25672_TCP_PROTO
$DAGSTER_RABBITMQ_PORT_4369_TCP
$DAGSTER_RABBITMQ_PORT_4369_TCP_ADDR
$DAGSTER_RABBITMQ_PORT_4369_TCP_PORT
$DAGSTER_RABBITMQ_PORT_4369_TCP_PROTO
$DAGSTER_RABBITMQ_PORT_5672_TCP
$DAGSTER_RABBITMQ_PORT_5672_TCP_ADDR
$DAGSTER_RABBITMQ_PORT_5672_TCP_PORT
$DAGSTER_RABBITMQ_PORT_5672_TCP_PROTO
$DAGSTER_RABBITMQ_SERVICE_HOST
$DAGSTER_RABBITMQ_SERVICE_PORT
$DAGSTER_RABBITMQ_SERVICE_PORT_AMQP
$DAGSTER_RABBITMQ_SERVICE_PORT_DIST
$DAGSTER_RABBITMQ_SERVICE_PORT_EPMD
$DAGSTER_RABBITMQ_SERVICE_PORT_STATS
should there be any difference between the dagit, celery, and dagster images??
the only difference we have between the images is on the dagit image: we expose 3000 and running the entrypoint.sh
Copy code
#!/bin/sh
export DAGSTER_HOME=/opt/dagster/dagster_home

# This block may be omitted if not packaging a repository with cron schedules:
####################################################################################################
# see: <https://unix.stackexchange.com/a/453053> - fixes inflated hard link count
touch /etc/crontab /etc/cron.*/*

service cron start

# Add all schedules defined by the user
dagster schedule up
####################################################################################################

# Launch Dagit as a service
DAGSTER_HOME=/opt/dagster/dagster_home dagit -h 0.0.0.0 -p 3000
j

johann

06/23/2020, 6:41 PM
I think celery should be a different image, checking
😮 1
k

Kevin

06/23/2020, 6:42 PM
Please and thank you sir!
j

johann

06/23/2020, 6:50 PM
I was wrong, they can be the same image as long as they have all the libraries
k

Kevin

06/23/2020, 6:51 PM
okay so they use the same base image, contain all the same files, so where do I go from here?
j

johann

06/23/2020, 6:52 PM
It's possible the DAGSTER_K8S_INSTANCE_CONFIG_MAP variable was set to none on the pod, you could check it's value
k

Kevin

06/23/2020, 6:53 PM
Copy code
echo $DAGSTER_K8S_INSTANCE_CONFIG_MAP
dagster-instance
seems rather sparse, is this value configured with the helm charts? and if so wouldn't it be populated to the images defined?
j

johann

06/23/2020, 6:57 PM
How are you launching the run?
Via dagit running on the pod?
k

Kevin

06/23/2020, 6:59 PM
yes, i launch the run via the playground with:
Copy code
execution:
  celery-k8s:
    config:
      job_image: '<http://my_repo.com/image_name:latest|my_repo.com/image_name:latest>'
      job_namespace: 'some-namespace'
j

johann

06/23/2020, 7:01 PM
And dagit is being served by the pod? (As opposed to being started with 
dagit
in terminal)
k

Kevin

06/23/2020, 7:02 PM
correct
j

johann

06/23/2020, 7:10 PM
Could you send the result of 'instance details' at the top left in dagit
k

Kevin

06/23/2020, 7:19 PM
My pleasure
Copy code
Dagster 0.8.4
DagsterInstance components:

  Local Artifacts Storage:
     module: dagster.core.storage.root
     class: LocalArtifactStorage
     config:
       base_dir: /opt/dagster/dagster_home

  Run Storage:
     module: dagster_postgres.run_storage
     class: PostgresRunStorage
     config:
       postgres_db:
  db_name: test
  hostname: dagster-postgresql
  password:
    env: DAGSTER_PG_PASSWORD
  port: 5432
  username: test

  Event Log Storage:
     module: dagster_postgres.event_log
     class: PostgresEventLogStorage
     config:
       postgres_db:
  db_name: test
  hostname: dagster-postgresql
  password:
    env: DAGSTER_PG_PASSWORD
  port: 5432
  username: test

  Compute Log Manager:
     module: dagster.core.storage.local_compute_log_manager
     class: LocalComputeLogManager
     config:
       base_dir: /opt/dagster/dagster_home/storage

  Schedule Storage:
     module: dagster_postgres.schedule_storage
     class: PostgresScheduleStorage
     config:
       postgres_db:
  db_name: test
  hostname: dagster-postgresql
  password:
    env: DAGSTER_PG_PASSWORD
  port: 5432
  username: test

  Scheduler:
     module: dagster_cron.cron_scheduler
     class: SystemCronScheduler
     config:
       {}

  Run Launcher:
     module: dagster_k8s.launcher
     class: CeleryK8sRunLauncher
     config:
       backend: amqp
broker: <pyamqp://test:test@dagster-rabbitmq:5672//>
dagster_home:
  env: DAGSTER_HOME
instance_config_map:
  env: DAGSTER_K8S_INSTANCE_CONFIG_MAP
postgres_password_secret:
  env: DAGSTER_K8S_PG_PASSWORD_SECRET

  Dagit:
     NoneType

  Telemetry:
     NoneType
j

johann

06/23/2020, 7:51 PM
I'm going to try to reproduce with your same config. In the meantime, could you check that the env variables are also present in the run job
k

Kevin

06/23/2020, 8:20 PM
that's going to be difficult as you cannot exec into a container in a completed pod. I can provide the logs from said pod, as i have before?
j

johann

06/23/2020, 8:23 PM
Sure, that could be helpful
k

Kevin

06/23/2020, 8:26 PM
eh voilà:
Copy code
Traceback (most recent call last):
  File "/usr/local/bin/dagster-graphql", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/cli.py", line 219, in main
    cli(obj={})  # pylint:disable=E1120
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/cli.py", line 211, in ui
    execute_query_from_cli(workspace, query, variables, output)
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/cli.py", line 96, in execute_query_from_cli
    workspace, query, variables=seven.json.loads(variables) if variables else None
  File "/usr/local/lib/python3.7/site-packages/dagster_graphql/cli.py", line 47, in execute_query
    else DagsterInstance.get()
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 239, in get
    return DagsterInstance.from_config(_dagster_home())
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 261, in from_config
    return DagsterInstance.from_ref(instance_ref)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 274, in from_ref
    run_launcher=instance_ref.run_launcher,
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/ref.py", line 205, in run_launcher
    return self.run_launcher_data.rehydrate() if self.run_launcher_data else None
  File "/usr/local/lib/python3.7/site-packages/dagster/serdes/__init__.py", line 351, in rehydrate
    config_dict,
dagster.core.errors.DagsterInvalidConfigError: Errors whilst loading configuration for {'instance_config_map': Field(<dagster.config.source.StringSourceType object at 0x7f242d4a6f50>, default=@, is_required=True), 'postgres_password_secret': Field(<dagster.config.source.StringSourceType object at 0x7f242d4a6f50>, default=@, is_required=True), 'dagster_home': Field(<dagster.config.source.StringSourceType object at 0x7f242d4a6f50>, default=/opt/dagster/dagster_home, is_required=False), 'load_incluster_config': Field(<dagster.config.config_type.Bool object at 0x7f242f383990>, default=True, is_required=False), 'kubeconfig_file': Field(<dagster.config.config_type.Noneable object at 0x7f2410824c90>, default=None, is_required=False), 'broker': Field(<dagster.config.config_type.Noneable object at 0x7f242b653ad0>, default=@, is_required=False), 'backend': Field(<dagster.config.config_type.Noneable object at 0x7f242b653b50>, default=rpc://, is_required=False), 'include': Field(<dagster.config.config_type.Array object at 0x7f242b5f5190>, default=@, is_required=False), 'config_source': Field(<dagster.config.config_type.Noneable object at 0x7f242b5f59d0>, default=@, is_required=False), 'retries': Field(<dagster.config.field_utils.Selector object at 0x7f242f327910>, default={'enabled': {}}, is_required=False)}.
    Error 1: Post processing at path root:instance_config_map of original value {'env': 'DAGSTER_K8S_INSTANCE_CONFIG_MAP'} failed:
(PostProcessingError) - dagster.config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_K8S_INSTANCE_CONFIG_MAP" which is not set. In order for this execution to succeed it must be set in this environment.

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/config/post_process.py", line 72, in _post_process
    new_value = context.config_type.post_process(config_value)
  File "/usr/local/lib/python3.7/site-packages/dagster/config/source.py", line 42, in post_process
    return str(_ensure_env_variable(cfg))
  File "/usr/local/lib/python3.7/site-packages/dagster/config/source.py", line 23, in _ensure_env_variable
    ).format(var=var)

    Error 2: Post processing at path root:postgres_password_secret of original value {'env': 'DAGSTER_K8S_PG_PASSWORD_SECRET'} failed:
(PostProcessingError) - dagster.config.errors.PostProcessingError: You have attempted to fetch the environment variable "DAGSTER_K8S_PG_PASSWORD_SECRET" which is not set. In order for this execution to succeed it must be set in this environment.

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/config/post_process.py", line 72, in _post_process
    new_value = context.config_type.post_process(config_value)
  File "/usr/local/lib/python3.7/site-packages/dagster/config/source.py", line 42, in post_process
    return str(_ensure_env_variable(cfg))
  File "/usr/local/lib/python3.7/site-packages/dagster/config/source.py", line 23, in _ensure_env_variable
    ).format(var=var)
j

johann

06/23/2020, 9:32 PM
And to clarify the same error isn't in the dagit pod logs?
You mentioned that you're calling entrypoint.sh for dagit? With helm that should already be taken care of by https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-k8s/helm/dagster/templates/deployment-dagit.yaml#L49
Otherwise, I think it would help me debug faster if you could upload your files to a repo. Alternatively you could try deploying https://github.com/dagster-io/dagster/tree/master/python_modules/dagster-test/dagster_test/test_project
k

Kevin

06/23/2020, 10:46 PM
k logs -n namespace dagster-dagit
Copy code
Starting periodic command scheduler: cron.
Usage: dagster schedule up [OPTIONS]

Error: There are no schedules defined for repository test_repository.

  Telemetry:

  As an open source project, we collect usage statistics to inform development priorities. For more
  information, read <https://docs.dagster.io/docs/install/telemetry>.

  We will not see or store solid definitions, pipeline definitions, modes, resources, context, or
  any data that is processed within solids and pipelines.

  To opt-out, add the following to $DAGSTER_HOME/dagster.yaml, creating that file if necessary:

    telemetry:
      enabled: false


  Welcome to Dagster!

  If you have any questions or would like to engage with the Dagster team, please join us on Slack
  (<https://bit.ly/39dvSsF>).
and if that is the case with entrypoint.sh being unnecessary, i'll remove it from its docker image and rerun the pipeline, and i'll also get a repo so that you can see what files we are using!! I do really appreciate the help!
j

johann

06/23/2020, 11:08 PM
You’re welcome, thanks again for trying this out while it’s rough around the edges
k

Kevin

06/23/2020, 11:43 PM
I view Dagster like a diamond in the rough, excavation might be a hassle but I believe!! small little repo with our files that we are using to deploy: github.com/NixTheMan/dagster you'll be required to build the core image, but that's what we are trying to run with!
Hi @johann I didn't have any luck with the test build you recommended :s Instead, I was greeted with this error:
j

johann

06/25/2020, 5:35 PM
Are you installing with pip -e? If so then make rebuild_dagit
Also thanks for the repo, nothing obviously wrong but will take a closer look
k

Kevin

06/25/2020, 5:47 PM
I removed the "-e" with the exception of the test_pipeline :s and bless you for looking into the repo, you'll keep my keyboard from being imprinted on my forehead! 🙂
j

johann

06/25/2020, 6:08 PM
Haha, did that fix this error?
k

Kevin

06/25/2020, 6:16 PM
sadly no T.T i seem to now be coming up with a lot of errors due to incompatible versions when running "make dev_install" as well as "make rebuild_dagit". So i'm going to clean my machine and retry :s
j

johann

06/25/2020, 6:20 PM
Gotcha. I think the test project uses local versions of the packages whereas you had the prod versions from pypi for your repo
I should have mentioned that the test project needs a dev setup which can be in a separate virtualenv https://docs.dagster.io/docs/community/contributing.html
k

Kevin

06/25/2020, 6:22 PM
i've created a conda environment for it 🙂
👍 1
I feel like I'm going down a rabbit hole XD make dev_install from the root directory fails with this:
Copy code
make[1]: Leaving directory '/home/kevin/workspace/tdagster'
# Checking for prod installs - if any are listed below reinstall with 'pip -e'
! pip list --exclude-editable | grep -e dagster -e dagit
cd js_modules/dagit/; yarn install && yarn build-for-python
00h00m00s 0/0: : ERROR: [Errno 2] No such file or directory: 'install'
Makefile:96: recipe for target 'rebuild_dagit' failed
make: *** [rebuild_dagit] Error 1
j

johann

06/25/2020, 6:48 PM
Dang I'm not sure what would cause that
Setting up everything to use the test project might be work than it's worth. I'll try to reproduce using your repo tonight.
😀 1
Can you test this out: in your pipeline's playground, add
Copy code
execution:
  celery-k8s:
    config:
     ...
      env_config_maps:
      - "<DEPLOYMENT NAME>-dagster-pipeline-env"
kubectl get configmaps
and grab whichever ends with 
pipeline-env
k

Kevin

06/25/2020, 10:15 PM
@johann we've been trying with the configmap for the celery workers and got a similar error to what i tried with dagster-pipeline-env:
Copy code
dagster.core.errors.DagsterUnmetExecutorRequirementsError: You have attempted to use an executor that uses multiple processes while using system storage in_memory which does not persist intermediates. This means there would be no way to move data between different processes. Please configure your pipeline in the storage config section to use persistent system storage such as the filesystem.
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/context_creation_pipeline.py", line 164, in pipeline_initialization_event_generator
    executor_config = create_executor_config(context_creation_data)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/context_creation_pipeline.py", line 276, in create_executor_config
    instance=context_creation_data.instance,
  File "/usr/local/lib/python3.7/site-packages/dagster/core/execution/context_creation_pipeline.py", line 50, in construct_executor_config
    return executor_init_context.executor_def.executor_creation_fn(executor_init_context)
  File "/usr/local/lib/python3.7/site-packages/dagster_celery/executor_k8s.py", line 113, in celery_k8s_job_executor
    check_cross_process_constraints(init_context)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/executor.py", line 209, in check_cross_process_constraints
    _check_persistent_storage_requirement(init_context.system_storage_def)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/executor.py", line 234, in _check_persistent_storage_requirement
    ).format(storage_name=system_storage_def.name)
we're going to try a single celery worker and/or an s3 within our k8s instantiation and see how it goes
j

johann

06/25/2020, 10:33 PM
I think you want something like
Copy code
storage:
  s3:
    config:
      s3_bucket: "dagster-scratch-80542c2"
      s3_prefix: "dagster-k8s-test"
In the playground
k

Kevin

06/29/2020, 6:04 PM
Hi @johann 🙂 We've implemented an s3 bucket with minio, however our pipeline still seems to be failing:
Copy code
Step add_one.compute finished without success or failure event, assuming failure.
logs from kubernetes show that it fails to find the pipeline 😕 any ideas?
j

johann

06/29/2020, 6:09 PM
Could you post the logs?
k

Kevin

06/29/2020, 6:12 PM
Copy code
2020-06-29 16:06:04 - dagster - DEBUG - math - 51c3700b-309b-43e5-b82f-800ec57c31a6 - PIPELINE_START - Started execution of pipeline "math".
dagster/solid_selection = "*"
2020-06-29 16:06:04 - dagster - DEBUG - math - 51c3700b-309b-43e5-b82f-800ec57c31a6 - ENGINE_EVENT - Submitting celery task for step "add_one.compute" to queue "dagster".
dagster/solid_selection = "*"
 event_specific_data = {"error": null, "marker_end": null, "marker_start": "celery_queue_wait", "metadata_entries": []}
            step_key = "add_one.compute"
2020-06-29 16:06:11 - dagster - ERROR - system - 51c3700b-309b-43e5-b82f-800ec57c31a6 - Step add_one.compute finished without success or failure event, assuming failure.
dagster/solid_selection = "*"
2020-06-29 16:06:11 - dagster - INFO - system - 51c3700b-309b-43e5-b82f-800ec57c31a6 - Dependencies for step mult_two.compute failed: ['add_one.compute']. Not executing.
dagster/solid_selection = "*"
               solid = "mult_two"
    solid_definition = "mult_two"
            step_key = "mult_two.compute"
2020-06-29 16:06:11 - dagster - DEBUG - math - 51c3700b-309b-43e5-b82f-800ec57c31a6 - STEP_SKIPPED - Skipped execution of step "mult_two.compute".
dagster/solid_selection = "*"
               solid = "mult_two"
    solid_definition = "mult_two"
            step_key = "mult_two.compute"
2020-06-29 16:06:11 - dagster - ERROR - math - 51c3700b-309b-43e5-b82f-800ec57c31a6 - PIPELINE_FAILURE - Execution of pipeline "math" failed.
dagster/solid_selection = "*"
{"data": {"executeRunInProcess": {"__typename": "PythonError", "message": "dagster.core.errors.DagsterSubprocessError: During celery execution errors occurred in workers:\n[add_one.compute]: (DagsterGraphQLClientError) - dagster_graphql.client.mutations.DagsterGraphQLClientError: Pipeline \"math\" not found: Could not find Pipeline <<in_process>>.test_repository.math:\n\nStack Trace: \n  File \"/usr/local/lib/python3.7/site-packages/dagster_celery/engine.py\", line 91, in _core_celery_execution_loop\n    step_events = result.get()\n  File \"/usr/local/lib/python3.7/site-packages/celery/result.py\", line 217, in get\n    self.maybe_throw(callback=callback)\n  File \"/usr/local/lib/python3.7/site-packages/celery/result.py\", line 333, in maybe_throw\n    self.throw(value, self._to_remote_traceback(tb))\n  File \"/usr/local/lib/python3.7/site-packages/celery/result.py\", line 326, in throw\n    self.on_ready.throw(*args, **kwargs)\n  File \"/usr/local/lib/python3.7/site-packages/vine/promises.py\", line 244, in throw\n    reraise(type(exc), exc, tb)\n  File \"/usr/local/lib/python3.7/site-packages/vine/five.py\", line 195, in reraise\n    raise value\n\n", "stack": ["  File \"/usr/local/lib/python3.7/site-packages/dagster_graphql/implementation/utils.py\", line 14, in _fn\n    return fn(*args, **kwargs)\n", "  File \"/usr/local/lib/python3.7/site-packages/dagster_graphql/implementation/execution/execute_run_in_process.py\", line 28, in execute_run_in_graphql_process\n    graphene_info, repository_location_name, repository_name, run_id\n", "  File \"/usr/local/lib/python3.7/site-packages/dagster_graphql/implementation/execution/execute_run_in_process.py\", line 158, in _synchronously_execute_run_within_hosted_user_process\n    execute_run(recon_pipeline, pipeline_run, graphene_info.context.instance)\n", "  File \"/usr/local/lib/python3.7/site-packages/dagster/core/execution/api.py\", line 149, in execute_run\n    event_list = list(_execute_run_iterable)\n", "  File \"/usr/local/lib/python3.7/site-packages/dagster/core/execution/api.py\", line 640, in __iter__\n    retries=self.retries,\n", "  File \"/usr/local/lib/python3.7/site-packages/dagster/core/execution/api.py\", line 568, in _pipeline_execution_iterator\n    pipeline_context, execution_plan\n", "  File \"/usr/local/lib/python3.7/site-packages/dagster_celery/engine.py\", line 166, in _core_celery_execution_loop\n    subprocess_error_infos=list(step_errors.values()),\n"]}}}
from the dagster-stepjob:
it'd be my pleasure 🙂 from the dagster-run:
Copy code
{
  "data": {
    "executePlan": {
      "__typename": "PipelineNotFoundError",
      "message": "Could not find Pipeline <<in_process>>.test_repository.math",
      "pipelineName": "math"
    }
  }
}
j

johann

06/29/2020, 6:56 PM
Is your repo still up to date?
k

Kevin

06/29/2020, 7:34 PM
Hi @johann we've changed the image to genimind/dagster-exp-core:v0.0.1, it is available on dockerhub. That new image has some more code added to it, but the rest of the files remain the same
j

johann

06/29/2020, 7:36 PM
Is it the latest dagster version?
k

Kevin

06/29/2020, 7:39 PM
it should pull the latest versions of dagster upon install, so it is running with 0.8.5
👍 1
j

johann

06/29/2020, 9:09 PM
Sorry I didn't catch this earlier, your workspace.yaml in the repo has
Copy code
load_from:
  - python_file: celery_pipeline.py
load_from:
  - python_file: pipeline.py
and should be
Copy code
load_from:
  - python_file: celery_pipeline.py
  - python_file: pipeline.py
Maybe you already fixed this, since you were able to execute it from dagit
👍 1
k

Kevin

06/29/2020, 9:30 PM
😮 i definitely haven't pushed the latest version yet then :s because I remember fixing that already
j

johann

06/29/2020, 9:43 PM
https://github.com/dagster-io/dagster/blob/d9ced0a4f17e4327cb856d89868c136058d118d6/examples/multi_location/workspace.yaml#L3-L3 Can you add a location name to your workspace yaml like above, then in the pipeline execution config add
Copy code
execution:
  celery-k8s:
    config:
      repo_location_name: whatever_name_is_choosen
a

alex

06/29/2020, 9:44 PM
apologies that you have to do these ^ shenanigans, its a temporary workaround while we finish the last few pieces of major architectural overhaul
k

Kevin

06/29/2020, 9:53 PM
sure thing @johann, and no worries @alex; it'll be like Christmas when this doesn't throw an error at me when I run a pipeline!
@johann sadly the update to workspace.yaml yields the same error, except replace <<in_process>> with "basic_location" blob confounded
🎉 So my team got it working! congadagster Thank you to everyone that helped me on this thread! thankyou With a minor modification to source-code for dagster-aws, enabling s3 from minio; a lengthy yaml in the playground; and a fairly flat directory structure of pipelines, repos, workspace, etc files, directly accessible from DAGSTER_HOME; my kubernetes deployment now successfully executes pipelines!! 🎊 So this leaves me with a few questions: 1. From the docs "Deploying: Running Dagit as a service" the 'template' Dockerfile puts the pipeline files in '/opt/dagster/app/'. We tried this and failed until we moved our pipeline files and repos into the DAGSTER_HOME directory. What is/should the '/opt/dagster/app/' directory be used for? 2. In the workspace documentation, I've noticed both 'attribute' and 'location_name'? What are these used for? 'location_name' seems like a lookup for the execution configuration, given what I have in my playground. So then what is 'attribute' for? 3. Having pipelines and repos included in the images deployed works. However, is it possible to have the workspace.yaml point to repos hosted on another s3 bucket? Rather than having the pipelines and repos directly built into the images? 4. Dagit is awesome! It's a great GUI, however I and my team do find ourselves staring a lot more at command line prompts than anything else. How does the dagster CLI operate with a K8s deployment? 5. As an add-on to 3, currently to run pipelines within dagit, we are supplying it with a lenghty yaml in the playground. I know from the tutorials I can pass a yaml file to the CLI, however, is there a way to make a lot of these the defaults passed? Could these be defined in the dagster.yaml? And if so, in a K8s deployment can I edit the configmap-instance.yaml file (from 'dagster-k8s/helm/dagster/templates/') to get the same affect? 6. Any prizes for longest thread? 😉 My team really likes dagster! We've seen significant improvements in processing speeds for our pipelines (in other deployments), where other products failed completely! So kudos on creating such an awesome product!
a

alex

06/30/2020, 2:39 PM
1. The more important bit is what the WORKDIR of the Dockerfile is since that will determine the current working directory when we invoke 
dagster
commands in the container. You can read more about 
/opt/
here https://www.pathname.com/fhs/pub/fhs-2.3.html 2. 
attribute
is the thing we will go look up in the file / module to find the 
@repository
. When 
attribute
is not specified we iterate over every attribute and grab everything thats an instance of 
@repository
. 
location_name
allows you to specify the 
repository_location_name
which is a property mostly hidden from users at this time (except for the workaround you encountered). 3. Theoretically sure - you need the code to end up in the container so it can execute it but if you want to have the 
@repository
function download stuff from somewhere that is technically possible. It will likely be pretty complicated to pull this off. Things that make this hard include not changing versions of the code at the wrong times, ensuring you have the right dependencies, and latency / caching trade-offs. 4. Running the CLI via a deployed container should work pretty well. Running the CLI locally could work if you configure your instance (
dagster.yaml
) and kubernetes CLIs correctly and have access to all the deployed components from your local. 5. You are looking for 
PresetDefinition
https://docs.dagster.io/docs/apidocs/pipeline#dagster.PresetDefinition 6. ❤️
k

Kevin

06/30/2020, 4:03 PM
hi @alex 🙂 1. That's interesting as in our image, we had set the WORKDIR as the '/opt/dagster/app/' which is where we had placed all the pipeline files. 2. Okay that's along the lines of what I assumed, thank you! 3. Is there any appetite to support this kind of functionality in the future? I would imagine my team wouldn't be the only people that would want this kind of functionality. 4. Awesome! 5. And awesome! 6. Thanks 😄
a

alex

06/30/2020, 4:07 PM
1. hmm - would have to dig in deeper to debug what was going awry 3. There will likely be some sort of 
git
based 
load_from
target at some point in the future which may be close to what you are looking for.
k

Kevin

06/30/2020, 4:11 PM
oh that would be beautiful!
one last question, we've had to do some custom-sauce with dagster-aws to support minio. Are there any plans to support minio for S3?
n

nate

06/30/2020, 5:00 PM
ha, what do you mean by custom-sauce? I would definitely be interested in making dagster-aws more compatible
a

alex

06/30/2020, 5:01 PM
ya do you have your code changes somewhere you can share them?
k

Kevin

06/30/2020, 6:28 PM
Indeed I do gentlemen! so we did a
Copy code
pip install -e dagster-aws
after replacing these files (attached) in the s3 directory under dagster-aws. The changes were just to pass the endpoint_url.
intermediate_store.pyfile_cache.pyobject_store.pysystem_storage.py
s

Simon Späti

10/29/2020, 1:20 PM
@alex @Kevin @nate little follow up, is 
minio-s3
already supported for 
storage
in core dagster, or would I need to apply above objects? 🤔
k

Kevin

10/29/2020, 3:19 PM
Hey @Simon Späti I haven't tested the latest version of dagster to check this. The last version I had deployed to my k8s was 0.9.2, and I required dagster-aws to work with 
minio-s3
but that version still required some minor code changes, for how my work wanted to operate with 
minio-s3
within dagster-aws
s

Simon Späti

10/29/2020, 3:30 PM
thanks kevin for the reply. I'm on it and it seems some clases got changed like 
ObjectStoreIntermediateStorage
. I'm trying to get it working. Hope so, otherwise I cannot run any pipeline as they cannot communicate to each other 😅
thanks so much @Kevin! Here would be the working one with version 
0.9.13
(mainly renaming of files and classes) Would be nice to bring this into the core dagster_aws. @nate? 🤔
intermediate_storage.pyfile_cache.pyobject_store.pysystem_storage.py
s

sandy

10/30/2020, 1:25 AM
passing the endpoint URL definitely seems like something we should enable. I thought it was possible through the s3_resource, but maybe I'm mistaken? @Simon Späti if you're interested in submitting a PR I'd be happy to review
👍 2
8 Views
Powered by