Hello guys, I've deployed Dagster with K8s, but when I launch a job it starts but it never runs. It keeps 'starting' for ever. Sensors, backfill and schedulers are all healthy and running. Am I missing something ? Thanks !

Hi Team — For RunRequest’s run_config parameter, is there a best practice for combining a config_from_files & a defined config:

RunRequest(run_config={config_from_files(...), 
                       "resources": {
                           "SOMETHING": { 
                                "config": {...}
                                        }
                                     }
                       })

Hi! Does anyone here load their DBT tables as individual ops? I'm wondering how the baked in DBT processor handles error scenarios - For instance, if a table fails halfway through the DAG, is there any way to resume from that table's failure, instead of rerunning the entire job

is there an example of submitting a pyspark job to dataproc? I only found one example of using

dagster_gcp.dataproc_op

under dagster-io, but nowhere else

How do i create 1-to-1 mapping between outputs and inputs of multi assets

Hi folks, I'm noticing a weird behavior on pretest configuration. We have a preset that we use for scheduled pipelines. One parameter we have in the preset is the calendar_time which uses datetime.utcnow(). I'm noticing that the pretest gets calculated only once after deployment (instead of getting calculated when the pipeline gets scheduled). For example, if the last time I deployed dagster is Oct. 8th, and a scheduled run for the pipeline triggered today Oct. 11th, pretest calendar_time will be supplied as Oct. 8th. My expectation was that it would be Oct. 11th. When I trigger a new deployment, it gets updated. My assumption is that pretest gets calculated only once during deployment, after that the values are fixed. Instead of getting calculated whenever a pipeline runs. We are migrating these pipelines/schedules/pretest from old version 12.5 and we didn't have this behavior. Our current version is 15.7 Anyone experienced this? any help or hint would be greatly appreciated

we're migrating from local spark to spark step launcher (preferably, currently weighing options) for pyspark on emr: it looks like the emr step launcher submit jobs to an existing emr cluster. does this means the step launcher doesn't create an emr cluster automatically if cluster id doesn't exist? so in this case I guess it would be ideal to create a script to auto start/stop the cluster during the day when it's mostly idle

Hi all, I’m still relatively new to dagster so this may be a silly question, but I’ve searched the docs and can’t see anything relating to this issue. I’ve setup a schedule to run daily. When running this using a dagster daemon, and reviewing using dagit, I can see the schedule is setup correctly and is toggled on. However, when watching the daemons logs, it seems like its running every 2 minutes (but not part of the schedule). I thought this may be a health check but it seems to be logging the runs of various ops of my graph. Can anyone explain why it seemingly runs every 2 minutes?

2022-10-11 18:22:28 +0100 - dagster.daemon.SensorDaemon - INFO - Not checking for any runs since no sensors have been started.

2022-10-11 18:23:23 +0100 - dagster.daemon.SchedulerDaemon - INFO - Checking for new runs for the following schedules: twitter_bot_graph_schedule

2022-10-11 18:23:23 +0100 - dagster.daemon.SchedulerDaemon - WARNING - Using UTC as the timezone for twitter_bot_graph_schedule as it did not specify an execution_timezone in its definition.

2022-10-11 18:23:23 +0100 - dagster.daemon.SchedulerDaemon - INFO - No new tick times to evaluate for twitter_bot_graph_schedule

2022-10-11 18:23:23 +0100 - dagster.daemon.SensorDaemon - INFO - Not checking for any runs since no sensors have been started.

2022-10-11 18:24:25 +0100 - dagster - DEBUG - twitter_bot_graph - faa141de-63f2-44d1-b695-101959254621 - 42154 - RUN_START - Started execution of run for "twitter_bot_graph".

2022-10-11 18:24:25 +0100 - dagster - DEBUG - twitter_bot_graph - faa141de-63f2-44d1-b695-101959254621 - 42154 - ENGINE_EVENT - Executing steps in process (pid: 42154)

2022-10-11 18:24:25 +0100 - dagster - DEBUG - twitter_bot_graph - faa141de-63f2-44d1-b695-101959254621 - 42154 - RESOURCE_INIT_STARTED - Starting initialization of resources [io_manager].

2022-10-11 18:24:25 +0100 - dagster - DEBUG - twitter_bot_graph - faa141de-63f2-44d1-b695-101959254621 - 42154 - RESOURCE_INIT_SUCCESS - Finished initialization of resources [io_manager].

2022-10-11 18:24:25 +0100 - dagster - DEBUG - twitter_bot_graph - faa141de-63f2-44d1-b695-101959254621 - 42154 - get_next_fixture_obj - LOGS_CAPTURED - Started capturing logs for step: get_next_fixture_obj.

Hi Dagster team! First, I wan't to say thank you for building such an amazingly intuitive and powerful framework, software defined assets for the win! I have been using the

build_asset_reconciliation_sensor

to keep some assets up to date as up-stream assets change, but one problem I ran into when deploying on K8s is that I can’t specify the tags for auto generated job the sensor kicks off. I’m using the K8sRunLauncher and I want to control the TTL of the K8s jobs that get spun up to run the sensors job. Would it be possible to expose the

tags

param in

build_asset_reconciliation_sensor

which would be added the the Job runs the sensor requests?

Hey there, is there a way to specify the run run launcher (or run launcher image) on a per-job basis? I ask because there are a few jobs which we would want to use a specific image for but not for other jobs.

👋 Hello, team! I deployed Dagster in Openshift and getting an issue while running a job.. I kind of understand the reason..But how do i overcome it? I am using the helm chart from the github..I tried to override the /opt/dagster to something else and the error remains with new folder..

dagster._core.errors.DagsterExecutionHandleOutputError: Error occurred while handling output "result" of step "multiply_the_word":
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_plan.py", line 224, in dagster_event_sequence_for_step
    for step_event in check.generator(step_events):
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 366, in core_dagster_event_sequence_for_step
    for evt in _type_check_and_store_output(step_context, user_event, input_lineage):
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 423, in _type_check_and_store_output
    for evt in _store_output(step_context, step_output_handle, output, input_lineage):
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 578, in _store_output
    handle_output_gen,
  File "/usr/local/lib/python3.7/site-packages/dagster/utils/_init__.py", line 432, in iterate_with_context
    return
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in _exit_
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/utils.py", line 82, in solid_execution_error_boundary
    ) from e
The above exception was caused by the following exception:
PermissionError: [Errno 13] Permission denied: '/opt/dagster'
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/utils.py", line 47, in solid_execution_error_boundary
    yield
  File "/usr/local/lib/python3.7/site-packages/dagster/utils/_init__.py", line 430, in iterate_with_context
    next_output = next(iterator)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/execution/plan/execute_step.py", line 557, in _gen_fn
    gen_output = output_manager.handle_output(output_context, output.value)
  File "/usr/local/lib/python3.7/site-packages/dagster/_core/storage/fs_io_manager.py", line 143, in handle_output
    mkdir_p(os.path.dirname(filepath))
  File "/usr/local/lib/python3.7/site-packages/dagster/utils/_init__.py", line 182, in mkdir_p
    os.makedirs(path)
  File "/usr/local/lib/python3.7/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)

I’m trying to write a failure hook that will drop up a temporary database when an op fails. The name of the temporary database is generated dynamically and returned as an output from the first op in the chain. Is there a way to make this name available somewhere on the HookContext, or another way to have it be accessible from within the hook function?

Hi there, quick question. I have an op with DynamicOutputs. Right now I'm being limited to 10 concurrent subprocesses in there. I added this to the dagster.config file and started up the daemon, but I'm still only getting 10 max subprocesses.

run_coordinator:
  module: dagster._core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 25

What am I missing?

Hello. Can you tell me please, is it possible to set relative date for

@run_status_sensor

decorator as well as implemented in

@schedule

decorator with

context.scheduled_execution_time

attribute? With

@schedule

I can use get previous date and pass it to config, like:

date = (context.scheduled_execution_time - timedelta(days=1)).strftime(DATE_FORMAT)

How can I do the same but with

@run_status_sensor

decorator?

Is there a way to combine sensors and schedules (e.g., execute job if AssetKey is available AND is 1st day of the month)?

Hi Dagster team, I recently upgraded the

dagster/dagster-k8s

image we are using from

0.14.9

to

1.0.12

. After running

dagster instance migrate

it seemed that all our ops, jobs, schedules etc. were all still intact and working fine. However, when browsing through the schedules interface I noticed that the

Partition Set

was no longer being displayed (given a

None

value) . I can still view all the partitions when I navigate to the run view of individual jobs + the

partition_set

column in the

runs

table seems to be filled as well. Are we missing some migration step for this particular field? Or maybe we need to add some additional config to the job in the repository?

Hello, Is there a way to make an optional resource key for an op? Essentially I have a test entrypoint for dagster which runs on local files, but the norm is for dagster to run using databases, etc. So, I want multiple ops to run if there is a value at a certain resource, but it should be possible for that value to be None.

This message was deleted.

Hi ! I use a

sensor

to detect files with high

minimum_interval_seconds

parameter (e.g. 12 hours) because the associated job performs many expensive queries on a database. In my daily use, I would like to be able to trigger the sensor tick manually (without having to wait 11 hours in the worst case) if I notice that an important file has been dropped. Is there a way to meet this need? I also wonder if it is possible to get the cursor of a sensor inside an op/asset to update it.

I have seen dagster talk at Italy, and the ata scientist team seems to be happy of dagster for heir ml pipeline (a graph was use to run the training pipeline and save the model as asset). I am wondering if you would recommend to use dragster for mlops and small teams ? (We are evaluating things like sage maker pipelines, airflow, dragster), so if you have some feedback or success story about dagster for mlops I will be happy to know more.

I’m getting this error and am confused:

dagster._core.errors.DagsterLaunchFailedError: Error during RPC setup for executing run: dagster._core.errors.DagsterRunNotFoundError: gRPC server could not load run 03e55140-2491-41b5-9560-496679471899 in order to execute it. Make sure that the gRPC server has access to your run storage.

Hey there! When you make changes to a workspace (or add a new one) what is the recommended approach to update the workspace? My concern is being able to update a workspace/repo without messing with currently running jobs.

hello. what’s the right pattern for defining a job that can be run on a variable asset? i.e., I have several assets of a similar type but different values and I want to run the same set of ops over all of them, and want to see the appropriate lineage of which materialization of an asset was used for a given execution of a job

hi, I am trying to pass input files (file key really) through a processing pipeline, generating the original and each intermediate step as an asset (with keys like file1_step1, file1_step2) so I have an ingest job that takes the file key and applies the graph, and the steps are called correctly, I'm just not sure how to use assets in this case would it make sense to use (dynamic) software defined assets at all? if yes, should I create assets using the API in each step op?

I'm looking at converting a few existing ops to assets. Two questions so far: 1. It doesn't look like assets support the

retry_policy

argument that ops do. Is there another way to add retries to asset materializations? 2. If I move an existing partitioned job to be a partitioned asset, I'm assuming I'll need to do some sort of manual migration to the database to keep materialization history? Or is there some magic way of keeping the names consistent to avoid that?

Hi, I am new to dagster. I had a query like is to possible to connect multiple jobs in dagster. Let's suppose we have a job that reads data and we have to pass the extracted data and transformed data to multiple separate jobs according to their functionality.

Hi, I have been using dagster for ETL automation in production. So far, we had dagster daemon not sending heartbeat problem. However, we didn't find any ERROR or CRASH in dagster daemon log. I found this pattern quite frequent in my daemon logs,

E1012 14:14:23.690197467 86 <http://fork_posix.cc:76]|fork_posix.cc:76]> Other threads are currently calling into gRPC, skipping fork() handlers

The last line of log is like this (attached images) which shows several dagster daemon with different pid, I think. My guess is the daemon create replica or duplicated process and that affect the heartbeat mechanism (not detected by dagit then the daemon is hanged/stuck/restarted/killed. Therefore, the job/pipeline is not triggered). Any keen ideas about this?

Hi, is it possible with the

dbt_build_op

to use the

--resource-type

parameter that is available in the

dbt build

cli command?

Hi there, I've been using [pip-tools](https://github.com/jazzband/pip-tools) for dependency management in my projects. I am trying to use it with Dagster, but it seems the packages

dagster

and

dagster-gcp

are incompatible. To reproduce the issue, create a file called

<http://requirements.in|requirements.in>

and list your dependencies in it. In my case,

dagster
dagster-gcp

After having installed pip-tools, run

pip-compile <http://requirements.in|requirements.in>

. I am getting the following error:

Could not find a version that matches grpcio<1.48.1,<2.0dev,>=1.32.0,>=1.33.2,>=1.38.1,>=1.43.0,>=1.49.1 (from dagster==1.0.12->-r <http://requirements.in|requirements.in> (line 1))
...

There are incompatible versions in the resolved dependencies:
  grpcio>=1.49.1 (from grpcio-status==1.49.1->google-api-core[grpc]==2.10.2->google-cloud-storage==2.5.0->dagster-gcp==0.16.12->-r <http://requirements.in|requirements.in> (line 2))
  grpcio<2.0dev,>=1.38.1 (from google-cloud-bigquery==2.34.4->dagster-gcp==0.16.12->-r <http://requirements.in|requirements.in> (line 2))
  grpcio<1.48.1,>=1.32.0 (from dagster==1.0.12->-r <http://requirements.in|requirements.in> (line 1))
  grpcio>=1.43.0 (from grpcio-health-checking==1.43.0->dagster==1.0.12->-r <http://requirements.in|requirements.in> (line 1))
  grpcio<2.0dev,>=1.33.2 (from google-api-core[grpc]==2.10.2->google-cloud-storage==2.5.0->dagster-gcp==0.16.12->-r <http://requirements.in|requirements.in> (line 2))

Is that a known issue? Any workarounds for that?

Hello, Maybe I missed something, but I saw that this MR was closed. Has the functionality it wanted to add been implemented in some other way?