Hello all, I'm trying to deploy Dagster for the first time and I Struggle setting the connection with my postgres db (on GCP). I set a Dockerfile and docker-compose, and I try to run it on my machine an I get the following message:

I'm seeing a strange bug (?) when running Dagster job on ECS using ECSRunLauncher. I have a set of software-defined assets. I start an asset materialization job from Dagit. Then exception is thrown inside the asset function. Exception is legit - there's an error in the input data, so I see step failure in the log in Dagit

dagster.core.errors.DagsterExecutionStepExecutionError: Error occurred while executing op...

. And here goes the most interesting part: underlying ECS jobs keep running and Dagster job also keeps running forever (Dagit shows it's status as

STARTED

), but nothing actually happens in this job, since it has already failed (but somehow Dagster doesn't know about it). Has anyone seen something like this before? Dagster version:

0.14.7

Is there a best practice for defining parents for an s3 stage in snowflake?

And related question: I have several thousands of queued runs which I know will hang in the same way. Is there a way to cancel/terminate them all without clicking though pages in Dagit and terminating them in batches of 25?

Hello all! Apologies in advanced if this is stated somewhere in the document (I couldn't find it). But I need help to schedule a job that has dependency on another job. For example, I have Job A that runs at 12AM daily, and Job B will run at 2AM daily, however, Job B should only run if Job A completed successfully as there are data which it depends from Job A. Could someone let me know how I could configure it in Dagster? Thanks in advance! cc @Kyle Downey

Hey guys, Running into some issues on 0.14.9 after upgrading from 0.14.6 when testing a dynamic output job w/ an inner graph locally (running dagit as a python process; no other infra). The same job works when deployed on K8s on 0.14.9. The job works locally on 0.14.6. Any thoughts?

dagster.check.CheckError: Invariant failed.
  File "/Users/chrisevans/Repositories/data-platform/dags/bi/.venv/lib/python3.9/site-packages/dagster/core/execution/plan/utils.py", line 47, in solid_execution_error_boundary
    yield
  File "/Users/chrisevans/Repositories/data-platform/dags/bi/.venv/lib/python3.9/site-packages/dagster/core/execution/plan/inputs.py", line 607, in _load_input_with_input_manager
    value = input_manager.load_input(context)
  File "/Users/chrisevans/Repositories/data-platform/dags/bi/.venv/lib/python3.9/site-packages/dagster/core/storage/fs_io_manager.py", line 152, in load_input
    context.add_input_metadata({"path": MetadataValue.path(os.path.abspath(filepath))})
  File "/Users/chrisevans/Repositories/data-platform/dags/bi/.venv/lib/python3.9/site-packages/dagster/core/execution/context/input.py", line 325, in add_input_metadata
    if self.asset_key:
  File "/Users/chrisevans/Repositories/data-platform/dags/bi/.venv/lib/python3.9/site-packages/dagster/core/execution/context/input.py", line 216, in asset_key
    check.invariant(len(matching_input_defs) == 1)
  File "/Users/chrisevans/Repositories/data-platform/dags/bi/.venv/lib/python3.9/site-packages/dagster/check/__init__.py", line 1167, in invariant
    raise CheckError("Invariant failed.")

I’m a newbie to dagster…so I’m running dagster 0.14.5 and am seeing this error: Sensor openlineage_sensor was started from a location repos.py that can no longer be found in the workspace, or has metadata that has changed since the sensor was started. You can turn off this sensor in the Dagit UI from the Status tab. Here is repo.py

from dagster import repository

from openlineage.dagster.sensor import openlineage_sensor

from hello_cereal import hello_cereal_job

@repository

_def_ hello_cereal_repository():

openlineage_sensor_def = openlineage_sensor(

_minimum_interval_seconds_=60,

_record_filter_limit_=60,

)

return [hello_cereal_job, openlineage_sensor_def]

Thoughts?

Hi! We’ve been running Dagster for a few months now, mostly to kick of dbt. Last week we introduced a new job that needs to run every 15 minutes, and I’m seeing a lot of failures happening - for some reason the job can not start and is killed by the run monitoring after the timeout. There’s really nothing in the logs, and there should be enough capacity on the k8s cluster. What can I do to debug this? Any pointers?

Why is ``@asset( compute_kind="my_kind",

only displayed in the job view of dagit and not in the asset graph? Is this a bug? I would like to see these tags show up in both views.

I'm integrating our dbt project with dagsters SDA using

load_assets_from_dbt_project.

Shouldn't dbt seeds be treated as dagster assets too?

Has anyone noticed that ops seem to take a bit longer to start (~5s) compared to solids which were instant? https://github.com/dagster-io/dagster/discussions/7338

Is there a way to stop a job mid execution (if a certain condition within one op is met) even if there are other ops branching out in the dag?

Good afternoon, Is there any way to control the synchronies? I run several functions within an graph and I have the problem that it does not wait for one to finish before starting the other. Any solution? Thanks! Example: @graph def do_some_task(): function1() -- (can be an external function, op or another graph) function2() -- (can be an external function, op or another graph) * Dagster starts to perform function2 before function1 has finished.

I have a multi-model IO manager for duckdb (parquet files) which can work via pandas and pyspark. For some ingest process I know for sure that I only want to use pandas (and not the pyspark resource). Currently, I must list pyspark under the

required_resource_keys

. As a result, pyspark must be instanciated first (and this costs some overhead). I s there any way (without making a 2nd copy of the multimodal IO manager without the pyspark resource to get rid of this delay? When trying to pass

"pyspark": None,

dagster complains. So far, it looks for me like I need to specify a dummy resource instead. Is this the right way to go?

I think I might have found a bug in dagit. When trying to stop a running sensor in dagit I get: TypeError: stop_sensor() takes 3 positional arguments but 4 were given.

Hi everyone, i use the graphql api to launch dagster runs from an api. I was wondering if is it possible to attach tags to the run. I wasn't able to find any reference in the docs! Thanks.

Are there any examples of how to partitioned job built by an asset group?

How can I specify the output type of an asset? Currently, I output a pandas dataframe, but

context.dagster_type.typing_type

is of type

any

as the asset seems to be not directly consumed by a downstream asset.

Thanks, @daniel for your help yesterday. I’m trying to integrate the openlineage-dagster plugin into my local dagster, which @daniel helped me achieve. All seems good on the dagster side, but when I run a job, I see the following in the daemon’s shell:

Sensor openlineage_sensor skipped: Last cursor: {"last_storage_id": 9, "running_pipelines": {"97e2efdf-9499-4ffd-8528-d7fea5b9362c": {"running_steps": {}, "repository_name": "hello_cereal_repository"}}}

and no OpenLineage event is generated. I’m not sure if this is dagster itself or the plugin. I’ve attached my repos.py, hello_cereal.py, serial_job.py and workspace.yaml files. Any thoughts??

https://docs.dagster.io/concepts/assets/asset-materializations#linking-assets-to-an-output-definition- sounds pretty good and interesting and I'd like to try it but then for a partitioned asset, how would this work? `Output`doesn't have a partition argument (https://docs.dagster.io/_apidocs/ops#dagster.Output)

Out

does (https://docs.dagster.io/_apidocs/ops#dagster.Out) but that seems like I'd be hardcoding it to a fixed partition (I don't really understand the usecase for this tbh). Is there something else I need to do to link an Op to a partitioned Asset?

Is it possible to extend the

@op

decorator to introduce some common setup code? I don't think a resource can do it since I want access to the op's

context

(and a single-process executor wouldn't call it for each op). I tried making a wrapping decorator but I get errors from Dagster's inspection of the argument lists:

def my_op(*args, **kwargs):
    """Dagster @op that opens a custom context manager based on some shared state."""
    def wrapper(func):
        @op(*args, **kwargs)
        def raw_op(context, *args, **kwrags): # my wrapper that does a little extra work
            with my_custom_setup(context):
                func(context, *args, **kwargs) # the original function that would normally be decorated with @op
        return raw_op
    return wrapper

How do I pass an argument to dagstermill op from another op? Code example in the thread.

Hello all - just starting my journey with dagster as a converted Airflow'er. Having an issue as soon as I start dagit. Operation name: InstanceSensorsQuery Message: (sqlite3.OperationalError) near "(": syntax error [SQL: SELECT subquery.id, subquery.selector_id, subquery.tick_body FROM (SELECT job_ticks.id AS id, job_ticks.selector_id AS selector_id, job_ticks.tick_body AS tick_body, rank() OVER (PARTITION BY job_ticks.selector_id ORDER BY job_ticks.timestamp DESC) AS rank FROM job_ticks WHERE job_ticks.selector_id IN (?)) AS subquery WHERE subquery.rank <= ? ORDER BY subquery.rank ASC] [parameters: ('03064b2df6239201293d653e3d171fd86ca400fa', 1)] (Background on this error at: https://sqlalche.me/e/14/e3q8) Path: ["repositoriesOrError","nodes",0,"sensors",0,"sensorState","ticks"] Locations: [{"line":152,"column":3}] This results in errors on schedules, sensors, and backfills pages not returning.

Is it possible to pass a

DailyPartitionsDefinition

to

load_assets_from_dbt_project

?

My jobs (which are supposed to spawn every 5 minutes and take ~1-3mins to complete) are stuck in “Starting” as far back as 3 days even though I have an autoscale GKE cluster I’m also seeing an extremely slow load then error when trying to go to this sensor page https://dagit.nautilusapp.xyz/workspace/example_repo@custom-example-user-code17/sensors/spawn_train_workers and this page https://dagit.nautilusapp.xyz/workspace/example_repo@custom-example-user-code17/sensors Do I need to beef up or repair my cluster? I’m still getting a bunch of completed runs even til now, so I know my sensor is functioning

is there something like

Permissive

that I can use in a

config_schema

except the only allowable keys are defined but not mandatory?

Hello! How can I kick off a partition backfill and specify config for resources?

Does anyone know how to use python assertion statements with dagster return variables? Just trying to validate a sql query before I push dbt to run it using dagster

@job
def my_pipeline():
    e = checkETL.check_sql_query()
    #assert e == True
    #run dbt Model

if __name__ == "__main__":
    result = my_pipeline.execute_in_process()
    # assert result.output_value ==True

In events, I see a

step_key

like

my_subgraph.opname

, but docs say it's deprecated. What's the right thing to use for a similar value? And does

context.solid_def.name

match that value? Including some notion of graph hierarchy in the key is useful.