Anyone ever see a google kubernetes cluster “repairing” and unavilable for > 24 hours? My runs started failing since 9:55am EST yesterday. I don’t think this is due to dagster, but hoping some GKE experts can help

Hi team, I am experiencing some strange behavior when trying to leverage the “externalConfigMap” setting to set the workspace for dagit. We are dynamically creating user deployments and would like to be able to load them into dagit, and I noticed on a separate thread that this setting was added recently to account for this use case. A screenshot of my dagit configuration is attached, where we point the workspace to an external configmap of "dagster-workspace". When bringing up the dagit/daemon pods, they are now both failing to start while saying they cannot locate the workspace file (see attached error screenshot). If I get the configmaps on the server and describe the dagster-workspace configmap, we can see that it is using the same configuration that worked when filling in the “servers” section of the values.yaml instead of the “externalConfigMap” section. Am I missing some type of volumeMount to get dagit/daemon to recognize the workspace?

Hi, We have our Deployments on Kubernetes, We use New Relic for Log Monitoring. What should be done from Dagster logging to capture logs only from

stdout

and

stderr

, I do not want to store logs in any place, What kind of logging manager should be applied in this case ?

Hi All, In kubernetes pod logs we are only able to see dagster event logs, but the application logs are not getting printed. However the application logs are visible in the dagit UI. we have done what is specified here :https://docs.dagster.io/concepts/logging/loggers#debugging-with-logs But the application logs was not visible in the kubernetes pod. can someone help??

Had a job fail overnight, and Dagster says this:

datagun_etl_gsheets (fac0512e-a510-4ab9-87f7-528dd3b81fa8) started a new run worker while the run was already in state DagsterRunStatus.STARTED. This most frequently happens when the run worker unexpectedly stops and is restarted by the cluster. Marking the run as failed.

Pretty sure I traced the event that lead to this in the GKE logs:

Scale-down: removing node gk3-dagster-cloud-nap-1jnp3gdh-282a8794-g2v8, utilization: {0.7951653944020356 0.2775837641283469 0 cpu 0.7951653944020356}, pods to reschedule: dagster-cloud/dagster-run-fac0512e-a510-4ab9-87f7-528dd3b81fa8-x27xz

I'm running GKE Autopilot with the multiprocess executor. Not sure why it decided to scale down the node in the middle of a run, but any idea? Is there a good way to handle this? I have retries enabled, but that errored for another reason I'll post in the thread

is there no way to specify custom labels for the Ingress' created by the helm chart? i see annotations

i see the general "dagster.labels" templte.. but there are some labels i need to specifically apply to ingress only and not chart wide... doesn't even look like I can apply labels on an ingress

is there a simple dagster user code kubernetes Deployment manifest example anywhere that doesn’t use Helm? I feel like I could figure this out on my own but if anyone’s got an example I’d love to see it and save some time.

Hey all, do any of the Dagster Images run off of Ubuntu 18.04? Specifically any of the default images inside the k8s helm chart.

How do we attache multiple packages into

dagsterApiGrpcArgs

dagsterApiGrpcArgs:
  - "--package-name"
  - "program_data.program_data"
  - "--package-name"
  - "survey_data.survey_data"

does not seem to be working

This is a followup question to one above. I am doing separate deployments for dagit/daemon and user-deployments. So if I want to add multiple repository into a single user-deployment what is the recommended way to achieve it via

dagsterApiGrpcArgs

? my single docker has code for multiple packages, each with it’s own

@repository

method

How can I specify a RunCoordinator for a k8s deployment of dagster? I have a QueuedRunCoordinator for my local testing setup that I need to translate over.

I am trying to use Helm (inside ArgoCD) to install dagster and dagster-user-deployments in kubernetes with Charts.yaml and seperate values.yaml files located in the same directory. Both Charts seem to be installing but the values.yaml files don't seem to be replacing the default values and still using the example code. I am using the files from the github repo and have tried many variations from full files to just whats needed. I can see the values being pulled in ArgoCD but they are not being used by dagster. I have several apps installed this same way and have no issues with them. I have a work around with embedding the values.yaml info in ArgoCD Application.yaml file but it would be more ideal to use the values.yaml files directly. Any ideas on what may cause the values.yaml to not be used? Your examples have Helm with a specified values.yaml file using the "-f" flag?

hey all! I am new to dagster and trying to understand something that should be simple but I can't figure out how to automate that. We are running dagster on k8s deploying the official helm chart. The core components are in the namespace

dagster

. All of our pipelines/jobs should run on specific namespaces rather than the dagster one. How do I configure the configMap

user-env

to be deployed and created within each namespace where the jobs must run? What is happening is that once I trigger a pipeline, the job attempts to start on the target namespace but then I get an error:

configmap "xxx-dagster-dagster-user-deployments-yyy-user-env" not found: CreateContainerConfigErro

Helm/K8s deploy question: Is it possible to change the namespace in the

CeleryK8sRunLauncher

? I am getting 403 errors when attempting to run jobs and it appears it’s trying to create jobs in the

default

namespace.

Attempting to deploy Dagster via a Helm chart to K8s, but receiving this error/stacktrace:

WARNING - Could not load location data-team-dag-code to check for schedules due to the following error: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/workspace.py", line 131, in _load_location
    location = self._create_location_from_origin(origin)
  File "/usr/local/lib/python3.7/site-packages/dagster/_daemon/workspace.py", line 150, in _create_location_from_origin
    return origin.create_location()
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/host_representation/origin.py", line 333, in create_location
    return GrpcServerRepositoryLocation(self)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/host_representation/repository_location.py", line 561, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File "/usr/local/lib/python3.7/site-packages/dagster/_api/list_repositories.py", line 19, in sync_list_repositories_grpc
    api_client.list_repositories(),
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 169, in list_repositories
    res = self._query("ListRepositories", api_pb2.ListRepositoriesRequest)
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 115, in _query
    raise DagsterUserCodeUnreachableError("Could not reach user code server") from e

The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1664381076.964556226","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3260,"referenced_errors":[{"created":"@1664381076.964555536","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"
>

Stack Trace:
  File "/usr/local/lib/python3.7/site-packages/dagster/_grpc/client.py", line 112, in _query
    response = getattr(stub, method)(request_type(**kwargs), timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)

My Dockerfile is simple

FROM python:3.7-slim

RUN pip install dagster dagit dagster-postgres


COPY . /

My helm values:

dagster-user-deployments:
  enabled: true
  deployments:
    - name: "data-team-dag-code"
      image:
        repository: "redacted/data-dagster"
        tag: 9282022v2
        pullPolicy: Always
      dagsterApiGrpcArgs:
        - "--python-file"
        - "./src/test_repo.py"
      port: 3030

I’m going to run this by my infra team as well, but curious if there was something else I should 👀

I am curious how my fellow Dagster-deployed-via-Helm-to-K8s people handle local development.

I think we all need a Vídeo deploying dagster on K8 using Docker as the example from dagster.io Something like deploying to Openshift would do, It's really hard to someone who does not know a lot about K8 to deploy the user code, the daemon and dagit

This message was deleted.

When deploying with Helm to K8s, is there a way to specify multiple repos, or point the

dagsterApiGrpcArgs

to the workspace YAML? Edit: Locally I’m able to specify multiple repositories in the workspace.yaml, but curious how to mimic that when deploying via Helm.

not sure if this should be asked in #dagster-support or here but was wondering if theres gonna be a

k8s_job_op

as an

@asset

(

k8s_job_asset

)https://docs.dagster.io/_apidocs/libraries/dagster-k8s#ops

K8s/Helm Question: Deploying via Helm, Dagster Dag Code has to be included as a repository , but this Airbyte/Dagster example doesn’t include a repository (just individual ops/jobs). Again, this might be my ignorance so far in Dagster fundamentals, but curious how to deploy DAG code that isn’t included in a “repo”.

ever get this resolved?

I’m curious how my fellow K8s/Helm deployers troubleshoot erroring

dagster-user-deployments.deployments

The web UI only shows the error stating hat it can’t connect to the gRPC server, but curious the best way to surface the exception that is causing the server to fail, which I do now by

kubectl log <pod-name>

Thanks!

How would I change the configuration of a dagster instance already deployed in k8s? I tried doing helm upgrade but that didn’t change the configuration shown in dagit.

Can I add a toleration to the Dagster agent helm chart so it can run in a Fargate-only EKS cluster? I'm new to Kubernetes, EKS, Helm, etc, so I could definitely be heading down the wrong path. But, I'm creating a k8s cluster in EKS with only a Fargate profile. When I install the Dagster agent Helm chart, the pod remains pending / unscheduled because it has a

<http://eks.amazonaws.com/compute-type=fargate:NoSchedule|eks.amazonaws.com/compute-type=fargate:NoSchedule>

taint. I think this is a similar issue. Is it straightforward / the right solution to just add a toleration to the agent, and if so how would I do that? Or is that a change that needs to be built into the chart?