hello, is there a way to specify memory / cpu requests limits for pods when using K8sRunLauncher ? this would be useful for us: this way the size of the kubernetes cluster would determine the concurrency level

Hi! I think the working directory change in 0.13.14 broke my k8s deployment (getting a “couldn't find module” error), how do i specify the working directory via helm? (I use dagster-k8s)

Hi, I’m experiencing a sync issue between Dagit and the command line (and Dagit itself). The pipeline already succeeded but the timeline keeps ‘running’ indefinitely:

Is run id globally unique or just within a pipeline?

Is there a way to access to what schedule is running from inside an op of a run from Python?

Hey Dagster folks, we just deployed our first sensor and are getting these 2 errors alternating. What do these mean and how do we fix them?

grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.DEADLINE_EXCEEDED details = "Deadline Exceeded" debug_error_string = "{"created":"@1642196167.763819910","description":"Error received from peer ipv4:10.0.128.47:4000","file":"src/core/lib/surface/call.cc","file_line":1063,"grpc_message":"Deadline Exceeded","grpc_status":4}" >
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 494, in _process_api_request
    api_result = self._handle_api_request(request, instance, user_code_launcher)
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 371, in _handle_api_request
    serialized_sensor_data_or_error = client.external_sensor_execution(
  File "/dagster/dagster/grpc/client.py", line 293, in external_sensor_execution
    chunks = list(
  File "/dagster/dagster/grpc/client.py", line 118, in _streaming_query
    yield from response_stream
  File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 426, in __next__
    return self._next()
  File "/usr/local/lib/python3.8/site-packages/grpc/_channel.py", line 826, in _next
    raise self

and

dagster.core.errors.DagsterUserCodeUnreachableError: Timed out waiting for response to request: DagsterCloudApiRequest(request_id='7d4c9656-2daa-4729-8778-0edc072fa688', request_api=<DagsterCloudApi.GET_EXTERNAL_SENSOR_EXECUTION_DATA: 'GET_EXTERNAL_SENSOR_EXECUTION_DATA'>, request_args=SensorExecutionArgs(repository_origin=ExternalRepositoryOrigin(repository_location_origin=RegisteredRepositoryLocationOrigin(location_name='human_resources'), repository_name='human_resources_repository'), instance_ref=None, sensor_name='lever_s3_sensor', last_completion_time=None, last_run_key=None, cursor=None), expire_at=1642196769.66473)
  File "/dagster/dagster/daemon/sensor.py", line 236, in execute_sensor_iteration
    yield from _evaluate_sensor(
  File "/dagster/dagster/daemon/sensor.py", line 268, in _evaluate_sensor
    sensor_runtime_data = repo_location.get_external_sensor_execution_data(
  File "/ursula/ursula/user_code/workspace.py", line 404, in get_external_sensor_execution_data
    result = self.api_call(
  File "/ursula/ursula/user_code/workspace.py", line 205, in api_call
    return dagster_cloud_api_call(
  File "/ursula/ursula/user_code/workspace.py", line 65, in dagster_cloud_api_call
    for result in gen_dagster_cloud_api_call(
  File "/ursula/ursula/user_code/workspace.py", line 121, in gen_dagster_cloud_api_call
    raise DagsterUserCodeUnreachableError(

In a @run_failure_sensor context, is there a way to get the original stack trace of the exception? This below just indicates the name of the step that failed.

context.failure_event.message

How can I pass a pandas dataframe from previous job run to next execution/run of the same job?

Hello! I have been trying to setup dagster/dagit in docker following the example in the docs. I have managed to get the setup to run if I execute my code in the same container as dagit using this in `workspace.yaml`:

load_from:
  - python_package: my_package

and this

dagster.yaml

bit:

run_launcher:
  module: dagster.core.launcher
  class: DefaultRunLauncher

but when I try to use the

DockerRunLauncher

I end up hitting one of 2 errors.

# dagster.yaml
run_launcher:
  module: dagster_docker
  class: DockerRunLauncher
  config:
    env_vars:
      - DAGSTER_POSTGRES_USER
      - DAGSTER_POSTGRES_PASSWORD
      - DAGSTER_POSTGRES_DB
    network: dagsternet

and

# workspace.yaml
load_from:
  - grpc_server:
      host: dagster_user_code
      port: 4000
      location_name: "my_package"

with the above setup, I see

details = "DNS resolution failed for service: dagster_user_code:4000"

and if I change the network in dagster.yaml to

mystack_dagsternet

then I see

failed to connect to all addresses

in the dagit logs (the network name is the one shown by

docker network ls

scoped with the stack name). I am running all of this with

docker stack deploy -f mycomposefile.yml mystack

and I have a container for

dagit

, one for

dagster_daemon

, and one for

dagster_user_code

(and some other services). They are all on the same network so they can reach each other. I must be missing something, but the error message is a bit cryptic. This is dagster 0.13.14. Any suggestions would be welcome :)

Hello, wanted to surface this thread again about memoization workarounds https://dagster.slack.com/archives/C01U954MEER/p1641468741018900 Can I access my memoized io manager from an op and check for has_output in order to short circuit the op (i.e return a memoized result if one exists)

I’m trying to save log files to s3. Based on this, I created a

dagster.yaml

file in the project root folder with the following content:

compute_logs:
  module: dagster_aws.s3.compute_log_manager
  class: S3ComputeLogManager
  config:
    bucket: "<bucket>"
    prefix: "dagster-logs"
    use_ssl: False
    verify: False
    skip_empty_files: True

When I run

dagster job launch --job <job_name> --workspace <workspace_file_name>

dagster appears to ignore these settings: there are no errors and logs are not saved to the specified s3 bucket/prefix. Thanks.

Hi Team,

'dagster-k8s/config': {
        'job_spec_config': {
            'ttl_seconds_after_finished': 60
        },

Is it possible to add annotations in job_spec_config ?

{ "annotations" : "<http://argocd.argoproj.io/sync-options|argocd.argoproj.io/sync-options>": "Prune=false" }

Hi! Do run_keys need to be unique for RunRequests of one sensor, or also across RunRequests of all sensors?

Hi Team, I am looking for a way to specify a nodeSelector for k8s jobs, and not in code. I am setting up a dagster instance and I would like to enforce on which nodes jobs are running

Hey guys, I’ve done a bit of digging around but haven’t yet found a good way to deal with secrets locally vs in kubernetes. Loading secrets as env vars in the config seems like the best (secure) way, but this has a couple problems with it: 1. If running dagit locally, I need to ensure all env var references are fulfilled even if I only want to run/test one job a. to get around this I would maybe: i. need some scripting to fill all required secrets before dagit launches? ii. Or maybe target only one job at a time with dagit - Is this possible? 2. When running in kubernetes, even though I only run jobs within their own kubernetes job, it seems dagit still requires all env vars to be present before it will startup. a. Why is this the case? I don’t really want to load every single secrets I may need to reference in my jobs into the dagit pod. b. I would like to only reference them from within the kubernetes job’s pod (where I can add a

dagster-k8s/config

tag to attach them) c. also currently it means I would need to reload dagit each time I create a new job requiring new secrets, instead of just reloading the relevant dagster-user-deployment 3. Or, since my secrets currently live in AWS Secrets Manager - could it make sense to simply create an op to pull these as part of the graph? I think this might work well locally too. Hmm writing this out may have helped me find a solution 😄 Still interested to hear the dagster team’s take on this

Hi Team! I'd like to understand how Dagster uses a database for internal storage. Is it recommended to use external one for better reliability (instead of the one created by Helm chart)? Does Dagster make a lot of load on the database? Is it a good idea to use the same database for Dagster internal storage and for ETL jobs?

What is the difference between schedules and schedules built from partitioned jobs? Couldn't they all be schedules in which I access the date field with 

context.scheduled_execution_time

?

Hi Dagster team! Is it possible to define a default

op_retry_policy

on a graph, but still bypass it and immediately abort execution if a special exception is raised?

Hey all! I have a sensor that’s slightly exceeding the 1min runtime limit, due to a database call in the sensor. In our case, said database rarely updates (~few times a year), which has led us to the inelegant solution of hardcoding the result of the database call in the sensor code. But wondering if anyone’s had success with allowing the sensor daemon to access cached objects that can be created without a 1min timer on it? Or more generally, any thoughts on a better design for this system? Thank you!

Hi Team! We are deploying dagster user code deployment with argocd via helm but problem is when the argocd auto sync enabled the k8s-job not creating. So We tried with these below configs in pipeline tags for jobs, argocd auto sync issue solved. But we are getting more issues like when we ran kubectl get pods ,pods terminating but jobs looks good and after executing pipeline getting this error : dagster.core.errors.DagsterExecutionInterruptedError.

job_metadata {
"annotations" : "<http://argocd.argoproj.io/sync-options|argocd.argoproj.io/sync-options>": "Prune=false"
}

Hi Team, I am currently using dagster's

execute_in_process

API to process a graph with ~15 ops. I am observing significant performance drop when switching to a persistent dagster instance with SQLLite/Postgres based run & event log storages. The execution time increases to

4 seconds

which was earlier taking around

250ms

. The executor is still the in-process one, so I guess its DB writes that are causing this overhead. Is that expected? Below are screenshots for execution times.

Hi Team, I’m (apparently) running into the concurrency limit of the default SQLite database with a job/op that `DynamicOutput`s every row in a pandas DF (about 3000):

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked
[SQL: INSERT INTO event_logs (run_id, event, dagster_event_type, timestamp, step_key, asset_key, partition) VALUES (?, ?, ?, ?, ?, ?, ?)]

My use case is rather simple and I don’t need to keep track of all the logs Dagster saves by default. I tried unsetting

DAGSTER_HOME

but that just uses a temporary directory that hits the same problem. Is there a way to work around this without having to setup my own SQL db? Something like reducing the amount of events saved or preventing/limiting the

DynamicOutput

ops from running in parallel?

Hi - I have a question of using a local docker implementation (following the dagster example) but am having an issue. I am using a local version of docker to orchestrate runs and would like to store data (e.g. csv’s as backups for database insert fails) and custom python logs back to the host system. I have mounted a local drive to all of the 3 images (repo code, daemon, and dagit) and have tested and ensured I can ssh into the running container and interact with the mounted drive. In my docker-compose.yaml I have:

volumes:
  dagster_mnt:
    external:
      name: dagster_mnt

which dagster_mnt was created with

docker volume ...

Under each service in the docker-compose.yaml I have added:

volumes:
      - dagster_mnt:/mount

However, I cannot interact with the mounted volume from the temp containers that dagit orchestrates, and when I attempt to ssh in, I can see that the volume is not mounted. Has any one run into this issue before or know of any work-around?

Hey there, what's the process for getting onto Dagster Cloud in the near term during the Early Access phase?

Hey - I’m getting an error

An exception was thrown during execution that is likely a framework error, rather than an error in user code.
Error
dagster.core.errors.DagsterInvalidConfigError: Error in config for resource slack
Error 1: Value at path root:config:token must not be None. Expected "(String | { env: String })"

Now when I exec into the pod running the user-code-deployment and

$ echo $SLACK_TOKEN

I can retrieve it. My job looks like this (shortened for readability)…

tesco_ev_naming_prod_job = tesco_ev_naming.to_job(
    resource_defs={
        "slack": slack_resource.configured({"token": os.getenv("SLACK_TOKEN")}, description="Slack on failures to #data-alarms."),
        # other resource configs
    }
    ),
    hooks={
        slack_on_failure(channel="#data-alarms", message_fn=slack_failure_traceback,
                         dagit_base_url="<http://localhost:3000>")
    },
   # other configs 
)

In addition I’ve defined my envSecrets (passed them in as literals using kubectl) on values.yaml for my user-code-deployment.yaml

envSecrets:
      - name: dagster-aws-access-key-id 
      - name: dagster-aws-secret-access-key
      - name: dagster-slack

Any ideas? I would have thought if I can access the env variable

$SLACK_TOKEN

in the pod it should work?

Hey everyone. My team is working on adding some unit and integration tests to our fledgling Dagster implementation and I'm hoping for some advice/guidance/best practices when it comes to testing ops that run SQL commands We have a job that downloads a zip file, extracts the file from the zip that we want to work with, runs a Great Expectations suite against the file to check for data quality, uploads the file to S3, then runs a bunch of SQL commands in a Redshift cluster for ELT steps (we'll be setting up other stuff like Meltano and dbt later, but for now SQL does the trick), To set up the Redshift resource, I am using environment variables for the host, port, username, password, and database. I created a configured Redshift resource that my job uses as defined below:

from dagster_aws.redshift import redshift_resource


@configured(redshift_resource)
def prod_redshift(_):
    return {
        'host': {"env": "DAGSTER_RS_HOST"},
        'port': {"env": "DAGSTER_RS_PORT"},
        'user': {"env": "DAGSTER_RS_USERNAME"},
        'password': {"env": "DAGSTER_RS_PASSWORD"},
        'database': {"env": "DAGSTER_RS_DB"},
        'autocommit': True
    }

But I realized a problem with this approach: if I try to unit test my op and/or job, it's going to execute those SQL statements in the prod Redshift cluster. My assumption based on the existence of the

dagster.aws.redshift.fake_redshift_resource

resource is that I should use that for testing because it will run everything except the SQL statements themselves, allowing me to test everything except the SQL syntax. But I can't wrap my head around how I would actually handle that configuration within my jobs and ops. I remember reading a while ago about the concept of "modes" but it appears that documentation is now in legacy status - is there a similar construct in the new solid-less op world?

Hi! We need to execute jobs in different clusters due to security restrictions. Can I use k8s_job_executor for that somehow?

Hello! Is the Dagster team aware of segmentation faults in the Dagster daemon when using python 3.10 against a PostgreSQL database?

Hi Team, in Dagster-pyspark, are we storing the intermediate results in some staging bucket? if so, is there any way we can overcome this problem?

Hello, I need to send email based on some events like, job success, some ops success faire,info, job ending. I saw the version 0.12 release "provide sending email out of the box" but I see only email on failure. Is there any other methods ? https://docs.dagster.io/_modules/dagster/utils#make_email_on_run_failure_sensor