Anyone at KubeCon this week?

In the "Kubernetes Grant AWS permissions>" documentation, the link for "Create an IAM role and and attach IAM policies" seems to be stale. It redirects to a generic "What is EKS" page. This may be the correct URL: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html .

Hello, I am running in to the same issue described here: https://github.com/dagster-io/dagster/issues/8314 where a large graph results in an argument list that is too long for bash. I made sure we updated dagster to 1.0.14, and can confirm that the yaml job manifest is passing these as environment variables:

metadata:
  annotations:
    <http://kubernetes.io/psp|kubernetes.io/psp>: eks.privileged
  creationTimestamp: "2022-10-25T16:41:57Z"
  generateName: dagster-step-6527edf0609f7ecea00021a01b2e8350-
  labels:
    <http://app.kubernetes.io/component|app.kubernetes.io/component>: step_worker
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: dagster
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: dagster
    <http://app.kubernetes.io/part-of|app.kubernetes.io/part-of>: dagster
    <http://app.kubernetes.io/version|app.kubernetes.io/version>: 1.0.14
    controller-uid: e2588cee-1c16-44d9-ae70-cce7842a789d
    dagster/job: ASSET_JOB
    dagster/op: etl_finngen_gwas_sumstats.download_gwas_transform_upload_parque
    dagster/run-id: ff7fff0b-ffe8-4990-b40a-c0d1631fb399
    job-name: dagster-step-6527edf0609f7ecea00021a01b2e8350
  name: dagster-step-6527edf0609f7ecea00021a01b2e8350-tzfc8
  namespace: dagster
  ownerReferences:
  - apiVersion: batch/v1
    blockOwnerDeletion: true
    controller: true
    kind: Job
    name: dagster-step-6527edf0609f7ecea00021a01b2e8350
    uid: e2588cee-1c16-44d9-ae70-cce7842a789d
  resourceVersion: "204837656"
  uid: 347329df-ee12-427b-b6eb-e1e68a3ca8a8
spec:
  containers:
  - args:
    - dagster
    - api
    - execute_step
    env:
    - name: DAGSTER_EXECUTE_STEP_ARGS
      value: '{"__class__": "ExecuteStepArgs", "instance_ref": {"__class__": "InstanceRef",
        "compute_logs_data": {"__class__": "ConfigurableClassData", "class_name":
        "NoOpComputeLogManager", "config_yaml": "{}\n", "module_name": "dagster.core.storage.noop_compute_log_manager"},
        "custom_instance_class_data": null, "event_storage_data": {"__class__": "ConfigurableClassData",
        "class_name": "PostgresEventLogStorage", "config_yaml": "postgres_db:\n  db_name:
        test\n  hostname: dagster-postgresql\n  params: {}\n  password:\n    env:
        DAGSTER_PG_PASSWORD\n  port: 5432\n  username: test\n", "module_name": "dagster_postgres.event_log"},
        "local_artifact_storage_data": {"__class__": "ConfigurableClassData", "class_name":
        "LocalArtifactStorage", "config_yaml": "base_dir: /opt/dagster/dagster_home\n",
        "module_name": "dagster.core.storage.root"}, "run_coordinator_data": {"__class__":
        "ConfigurableClassData", "class_name": "QueuedRunCoordinator", "config_yaml":
        "{}\n", "module_name": "dagster.core.run_coordinator"}, "run_launcher_data":
        {"__class__": "ConfigurableClassData", "class_name": "K8sRunLauncher", "config_yaml":
        "dagster_home: /opt/dagster/dagster_home\nimage_pull_policy: Always\ninstance_config_map:
        dagster-instance\njob_namespace: dagster\nload_incluster_config: true\npostgres_password_secret:
        dagster-postgresql-secret\nservice_account_name: dagster\n", "module_name":
        "dagster_k8s"}, "run_storage_data": {"__class__": "ConfigurableClassData",
        "class_name": "PostgresRunStorage", "config_yaml": "postgres_db:\n  db_name:
        test\n  hostname: dagster-postgresql\n  params: {}\n  password:\n    env:
        DAGSTER_PG_PASSWORD\n  port: 5432\n  username: test\n", "module_name": "dagster_postgres.run_storage"},

This message was deleted.

Is it likely the dagster-agent Docker image is not runnable on amd64 AMIs? I happened to pick an amd64 instance class + AMI for an EKS node group, and am seeing a crash-on-launch for my agent pod, which looks similar to https://stackoverflow.com/questions/72042794/when-creating-pod-it-go-into-crashloopbackoff-logs-show-exec-usr-local-bin-do . Should be easy enough to switch to another instance/AMI type.

Hi! I wanted to ask a question about SDAs in relationship to distributed user code deployments. We have one central dagster daemon/dagit instance each, but multiple dagster user code deployments. I know SDAs in one repository can refer to input SDAs from another repository using SourceAsset, but is it possible for an SDA to refer to an input SDA in another user code deployment? I currently have a POC hooked up that does exactly this and it seems to be linking just fine in the graph view on Dagit (the input SDA is rendered as a source asset that links to another asset key), but when I go to run the downstream SDA it complains about there being

NoSuchKey

for the upstream asset.

Any advice on using fluentd/bit to upload JSON-formatted application logs from user code in k8s? The standard would be a daemonset, but that pulls from the container's stdout, which is a bit noisy. Our current (on-prem) setup is to have a separate fluentd process that tails a JSON-formatted log file. Is there support for a k8s agent to start multi-container pods for user code (I don't see anything in its Helm values)? Is it a terrible idea to just also run fluentbit in our application image?

Hello, I am creating an asset from a graph with with a

"dagster-k8s/config"

tag using

dagster.AssetsDefinition.from_graph

. I notice that this tag doesn't seem to get added to the run configuration. I want this asset materialization to run in a specific node group with specific resource requests. Will the job not pick up these tags?

Hi everyone! 👋 I am in the process of deploying Dagster on EKS and had a few questions I am hoping someone can help me with as I haven't found an answer myself 🙂 • I see a value called "rbacEnabled" in the values.yaml file, I assume that this has to do with Kubernetes RBAC and not user authentication / authorization in Dagit, right? I don't believe that authentication and authorization is available in the open-source Dagster (source: https://github.com/dagster-io/dagster/issues/2219) but correct me if I'm wrong as I would like to enable this if possible. • I would like my job pods to be deleted automatically after completion (or failure). I found the following documentation which mentions how to do this via Python code, and this method works. However, this means that I have to specify this for each job I create. I tried to add this configuration myself to the values.yaml file so that I don't have to specify it per job in my code but it is automatically applied to all jobs. Unfortunately I could not get it to work. Has someone been able to do this? Thanks in advance! If someone has any questions for me I'd be more than happy to answer 🙂

hello, I've got Dagster Hybrid running in EKS. I have multiple agents running in multiple namespaces on the same cluster. The idea behind this is that each branch deployment creates its own unique namespace, which has corresponding unique permissions in our other AWS resources (this way, while you test something on your branch deployment, you don't have permission to modify any other branch's AWS resources or the production resources). my problem is that when i execute a job in my branch deployment, Dagster seems to be arbitrarily picking one of the agents to run it with. so for example, my namespace is called

pr-560-dagster

, and it has a service account called

pr-560-worker

. there is another namespace,

pr-100-dagster

, with a similar service account. i've configured my dagster job to use

k8s_job_executor

. the code i have is something like this:

@job(
    name='Market_Insights_ETL_and_Export',
    executor_def=k8s_job_executor,
    config={
        ...
        'execution': {'config': {'job_namespace': 'pr-560-dagster', 'service_account_name': 'pr-560-dagster-worker'} }
    }
    ...
)

and this shows up in Dagit as well (see first image). but even with this config, the run gave me this error:

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch is forbidden: User \"system:serviceaccount:pr-100-dagster:pr-100-dagster-worker\" cannot create resource \"jobs\" in API group \"batch\" in the namespace \"pr-560-dagster\"","reason":"Forbidden","details":{"group":"batch","kind":"jobs"},"code":403}

it seems to have acknowledged the namespace that i passed it, but it's ignoring my service account specification and picking one arbitrarily. in the second and third images, you can see two runs of the same job with the same config - it picks different namespaces for them. it seems to alternate back and forth between each namespace. if it's important at all, the particular op that i'm trying to execute is also a kubernetes job, so i'm using

k8s_job_op

to configure that. and yes, i've confirmed i'm passing

pr-560-worker

as the service account for that as well.

Hi, I upgrade dagster kubernetes cluster from

0.13.4

to

0.14.15

and I got

dagster-daemon

pod in evicted status. Do you know what can cause that issue?

Question for anyone running dagster in kubernetes in production: what is your deployment process like?

Copy the question here for visibility.

Hey guys, anyone have any suggestions for this?

I was running a job that has a dbt op. I terminated the job, but I just noticed that dbt finished running the queries it was currently executing. This was problematic for me because we have post-hooks in dbt that alter the permissions of tables, so this meant the tables got replaced but then the post-hooks never ran, leaving the tables with the wrong permissions. when you run dbt on your local terminal, you can press Ctrl+C to issue a SIGINT. when dbt receives this, it cancels the queries it is running in Snowflake. is there any way i can configure my kubernetes pods to issue a SIGINT when i terminate a Dagster job?

is there a simple way to have jobs defined with an

dagster_k8s_executor

fall back to the default executor (multiprocess) when being run outside of an environmenet with K8SRunLauncher? I understandably get this error when trying to run locally.

dagster._core.errors.DagsterUnmetExecutorRequirementsError: This engine is only compatible with a K8sRunLauncher; configure the K8sRunLauncher on your instance to use it.

Where would I find the logs for the grpc server in the dagster-user-deployments pod?

I had a run fail because (I think) a step pod ate up all the node's memory and that killed the agent / job / other step pods when EKS dropped that node and created a new one. The error log I got in the job was "Agent 956008f6 (my-agent) detected that run worker failed: K8s job failed". I'll adjust the EC2 instance type for our nodes, and it wasn't too hard to track down, but reporting the details in case there's an opportunity to make the system more robust / errors more clear in cases like this.

Minor thing, but in Dagit it says

Executing step "my_op" in Kubernetes job dagster-step-89ab8bb9534b72ded96243830e8567be.

, but the pod name I need for

kubectl

is actually

dagster-step-89ab8bb9534b72ded96243830e8567be-cz9ts

with an extra suffix. Is there a way to get that suffix in the Dagster log message? It would be handy to be able to use that pod name directly in debugging commands.

is there a way to set

ttlSecondsAfterFinished

for the K8s jobs cluster wide? instead of having to annotate each

@job

?

Not sure if here is the right place to ask: We deploy OSS dagster on kubernetes, after upgraded to 1.1.6 (from 1.0.17) we see the

Definitions reloaded!

keep poping up every ~20 seconds. Rolling back to 1.1.5 resolve the issue, any idea why?

Hi all, I am trying to run a sensor that launches a job. I would like to give it an

executor_def

like when using a

@job

annotation. I yield a

RunRequest

, where should I put the executor definition? Is this documented somewhere?

Will a RetryPolicy on an

@op

cause a retry if the pod is OOMKilled? From the docs it looks like the retry is handling an exception from the op, and I'm not sure if that handling encapsulates the pod-termination check. I'm looking at an op that failed with "STEP_FAILURE Step my_step failed health check: Discovered failed Kubernetes job dagster-step-1b1e67c47da27410811b5606965fb1c4 for step my_step" and showed Status: OOMKilled in

kubectl describe

but ran fine on retry w/ plenty of RAM headroom as far as I can tell.

Is there some maximum time a pod is allowed to be pending before it's treated as failed? In my case with 5k dynamic outs, eventually some of the pods which were still Status:Pending according to kubectl showed up as failed in Dagit (stack in thread), which caused my job to fail. I'm trying to figure out if it's a reasonable theory that the agent just got tired of waiting for the pods to schedule (even though k8s thought they would eventually get scheduled) and decided they must have had an error, or if there was some error status or communication error on/in the pod I overlooked.

hey all, is there a way to add a nodeAffinity for Dagster jobs to run only on a specific node in the Helm Chart? Essentially, I'd like to have all

dagster-steps-*

running on a pool of nodes which has more resources available.

we've set the

job_namespace

value and Dagster seems to be reading the config correctly, but the jobs are still being created in the same namespace as the daemon is deployed. am I missing something? relevant config section.

run_launcher:
  module: dagster_k8s
  class: K8sRunLauncher
  config:
    job_namespace: dagster-jobs

My pods are erroring with "429 Client Error: Too Many Requests for url: https://formenergy.agent.dagster.cloud/graphql". I'm running about 70 step pods concurrently across 2 nodes, and each one only takes a couple seconds. I assume this means I'm getting rate limited on the GraphQL API side. Is there a way to get a higher quota?

I think I know the answer to this (not by default), but figured I’d ask here: Is it possible to configure dagster to create a new pv/pvc for each run of a job?

Team, currently if i need to add any new user-code-deployment, i am updating both the values.yaml & workspace.yaml and applying helm upgrade (dagster==0.14.15). Due to which both the dagster daemon, dagit are restarting (adding init containers for user-code lookup). So Is there any way i can segregate dagster daemon/ dagit from user-code deployment using helm? So that my already triggered user-code deployment using graphql queries will not fail (It should not affect already running user-code)?

I am curious the best way to exclude some Dagster/Airbyte stream configs from the application build and instead provide it as a ConfigMap. I am not experienced in using ConfigMaps or Volumes, but curious what the best method (and allowable using Helm deployment) to reference configuration yamls within the application code.