Hello there! I am trying to make a dependency between two graphs using the GraphDefinition. I see that I can define input mappings but what if I want Graph B to depend on graph A (all of the ops/graphs in graph A need to finish before a single op/graph in graph B starts.) how do I define that dependency?

Hi all, I have a question regarding

dagster_dbt

. We

load_assets_from_dbt_project()

and have all our dbt assets in Dagster. We can materialize (dbt) assets and everything works great! However we use

K8sRunLauncher

so asset materialization runs in a separate (ephemeral) pod. Each dbt asset materialization generates a

manifest.json

, saved locally on the pod. We want to grab this file and store in on our s3 bucket, however since the pod is ephemeral it’s killed before we can retrieve the file. What’s the best way to solve this?

Hi there, How can I retrieve results from materialize for one asset? What is the best way to test assets in Dagster?

# %%
from dagster import asset, materialize_to_memory

@asset(
    name='simple',
    key_prefix='prefix'
)
def simple() -> str:
    return "simple"

result = materialize_to_memory(
    [simple]
)

simple_res = result.output_for_node(simple.key)

This code throws an error

ParameterCheckError: Param "handle_str" is not a str. Got AssetKey(['prefix', 'simple']) which is type <class 'dagster._core.definitions.events.AssetKey'>.

What is the str key for this asset then? I used not to put a key_prefix and fetch asset results by just calling

result.output_for_node('simple')

Hello! 👋 How should I go about defining a multi_asset of 4 assets, of which only one (named

organisations

) has a dependency on an additional upstream asset? The upstream asset (named

organisations_stats

) may also be part of a different multi_asset definition. I see that

non_argument_deps

are only supported as a decorator argument (not an AssetOut argument) and is applied to all assets, which results in a lineage showing all 4 assets as downstream of the dependency, which would be incorrect. The docs on defining and constructing dependencies describe

op

definitions, but as I'm new to Dagster and have so far been working at the level of assets, I haven't been able to see how to apply that guidance to

asset

definitions. The real-world sequence I'm trying to represent in Dagster is this: 1. A single API call is made to a data warehouse to refresh 4 materialised views represented by the

outs

shown below, every 30 minutes 2. One of those views,

organisations

, depends on another materialised view

organisation_stats

which is refreshed once daily due to expensive computation but remains valid in that time 3. I would like to use Dagster to manage both of these refresh schedules, and to display the dependency between these two views 4. At the same time, it would be convenient to be able to define the other views (there may be several more) together using a multi_asset, or another approach, because they are all materialised as part of the same operation / API call What do you recommend? Many thanks for your help! 😀

@multi_asset(
    outs={
        "opportunities": AssetOut(),
        "contacts": AssetOut(),
        "projects": AssetOut(),
        "organisations": AssetOut(),  # <- has an upstream dependency...
    },
    # non_argument_deps={"organisations_stats"},  # ...which is a dep of organisations only, so this doesn't work
)
def multi_asset_example(context):
    # function continued...

@asset
def organisations_stats():
    """This is the additional asset upstream of organisations. It might be defined like this, or as part of a different multi_asset."""

Hi, I'm pretty new to dagster. Im deploying dagster via Azure/jfrog-artifactory to Openshift. Under helm i can see dagster(values.yml) and another dagster-user-deployment (value.yml). Since I want to user code seperated from daemon/dagit i have made enableSubchart: false and workspace:true in dagster(value.yml). Please suggest what else is required to have user code as seperate so i can add repos from my local and it doesn't trigger the deamon/dagit everytime i trigger the user code alone..🙂

Hey newbie here, is there an equivalent concept to airflow’s operators in dagster? I have a bunch of dockerized applications that run in ECS. Currently airflow triggers runs for these in ECS using the ECSOperator. What’s the best way to go about this in dagster? Would I create an

op

to trigger an ECS task and another to poll for completion? Is this something that already exists or do I need to “roll my own”?

Trying to learn about “build_asset_reconciliation_sensor” and possibly move away from cron-based job scheduling (as suggested here: https://docs.dagster.io/guides/dagster/scheduling-assets?_gl=1*1w38q5o*_ga*MTM4ODU0Mjk5OS4xNjcxNzI4MjUw*_ga_84VRQZG7TV*MTY3MTcyODI1MC4xLjEuMTY3MTcyOTA2Mi4wLjAuMA..) With cron-based job scheduling, I’m able to provide config to all assets that are materialized by the job. With build_asset_reconciliation_sensor this does not seem possible. So if this reconciliation sensor recognizes that some assets are in need of being refreshed, how does run config get provided to the assets? I’ve built my assets on the assumption that the best way to provide config to them is via the @asset config_schema param and would prefer to not have to move away from this (unless there is a much better way that is suggested and supported).

Hey there, are there any simple examples of running a

dbt

model inside of BigQuery using dagster?

Hi there, Is there a way to materialize single asset from the asset outs distionary in dagster. For ex. I am using

multi_asset

to define the asset outs and My asset defination is

[AssetKey(['dbname', 'TABLE1']), AssetKey(['dbname', 'TABLE2']), AssetKey(['dbname', 'TABLE3']), AssetKey(['dbname', 'TABLE4'])]

But I want to select only

AssetKey(['dbname', 'TABLE2'])

. When I try to do so, getting an error

AssetsDefinition does not support subsetting

Can you upgrade GRPCio? I am running into issues with other packages: • grpcio-tools 1.48.2 depends on grpcio>=1.48.2 • dagster 1.1.7 depends on grpcio<1.48.1 and >=1.32.0

Hello everyone. I start work with Dagster.Tell me where should I go. We use meltano instead of dagster. We have about 10 elt operations We need to run operations using graphQL That is, inside the graphQL request, we need to pass the configuration to run one of the 10 tasks. Who knows how this can be implemented? Here is how it should look like 1. Python code calls client.submit_job_execution 2. Transfer the result of execution to my Google cloud Function (I can call my cloudFunction via http request) 3. Transfer the result of the previous cloudFunction to the next cloudFunction. Is it possible to organize this?

Hi Everyone, I set up Dagster with my own user code using the single Helm Chart deployment and it worked fine. Now I tries separating both and I fail miserably... The user-code-deployment is setup and works. The dagster helm chart fails though. Both the dagit and dasgter-deamon pod fail with

Error: No arguments given and no [tool.dagster] block in pyproject.toml found.

To change from single helm chart to 2 different I just did 3 changes: • disabled dagster-user-deployment in dagster helm chart (and enableSubChart as well) • enabled the workspaces in dagster helm chart • added my workspace (port and service are correct) I am running on the latest helm chart versions: 1.1.7 Thanks for the help!

Another issue I am facing when trying to apply seperation of concerns. When I try to setup an external Postgres and reference the service name as host, it does not work. Should it be something else than the service name? Postgres is up an running, service as well and is tested. Secret and user are working as well. The dagit and dagster-deamon pod are stuck in

Containers with incomplete status: [check-db-ready]

In the logs it just shows:

\tdagster-postgresql:5432 - no response
waiting for database

Even though the service is there... Again, thanks for any input!

I know it is not a lot of information, But experiencing that some scripts that works fine plain python are running infinite in dagster ops. It has happen in different ops. Has Anyone experienced similar ?

Hi there...I'm sorry but I can't seem to figure this out. I have an op running in a job that materializes an asset. I have another job that I run subsequently and I want an op in that job to reference the asset I created in the first job. Is there any way to do that or am I just missing something?

It is possible to trigger the launch of Google cloud functions from dagster (I know that this is possible by calling the endpoint url), but is it possible to somehow catch all the logs returned by cloudFunction?

I'm thinking about checking out Dagster for some of our genomics/bioinformatics workflows. The Dagster model is clearly quite rich, and so it's not obvious to me what is the best way to map some of our problems to your DSL/API.

Here are a few issues.

Many (most?) bioinformatics tools are built to read and write to a local filesystem. For example: read a "fastq" file along with some extra reference datafiles and write out a "bam" file, all to local disk. How would this be encoded as an op? Take a set of paths as input and return a set of paths as output? Would you then rely on an IOManager to bring the files from S3 to local disk and back?

And in that case, what if I want to chain together a whole DAG of operations. It seems wasteful to read/write to S3 in between each op, when you could just continue the same op with the same local disk context instead. Is there a way to describe this?

Perhaps using a graph that has an S3 IOManager at the boundaries and local file system internally? This seems like it would be complex/messy though, no?

Also, I had a hard time understanding when I should be using an asset vs jobs/ops. In the case above, say, a new "sequencing run" is performed that generates some raw data files. Say the sequencing run is called "SR1" and I have it's identifier in advance. That sequencing run will produce, say, a dozen different raw data files corresponding to different sample names. Each of these raw data files has some transformation steps and finally they are processed jointly to generate a processed form of the data, again one file per sample. Each of these processed files ends up in a different google bucket under a subdirectory called "SR1". So what should be the assets? Is the "raw" "SR1" an asset? Or is each file inside of SR1 an asset? And same question for the processed files. Is the SR1 subdirectory in S3 the asset? Or each file inside of that subdirectory? Finally, the process of going from the raw files to the processed files produced a lot of intermediate files that do not need to be stored. How do I string together assets that contain a whole DAG of operations in between them?

Also, hi @sandy!

I did see the blog post by ZephyrAI, but it was short on details of how they express their operations.

A simpler question: we perform a sequencing run "SR1" using a certain instrument, which results in some raw data files in Illumina's "cloud". We have a workflow for downloading those files, pushing them to S3, and also writing some additional metadata into Notion. It would be great to have this run as a Dagster workflow. Again, it doesn't seem obvious to me what is the right way to map that to the Dagster API.

Any reason why setting

working_directory: my_working_directory

in the workspace.yaml does work? my code always seems to load from the location where dagit was started from

Hi all, I trie to deploy dagster with helm chart and an external postgres database. My dagster daemon and dagit pod still in init state. I use an another chart for the user code deployment. any ideas ?

I start my app using

dagit -f app.py

Is there a way to access the metadata from a DynamicOut in an IOManager?

Hi all! Iam trying to follow the docs on setting up a multi-asset sensor to run a job when any partitions have new materializations (https://docs.dagster.io/concepts/partitions-schedules-sensors/asset-sensors#when-any-partitions-have-new-materializations), but the method used, namely,

latest_materialization_records_by_partition_and_asset

, can only be used when the partitions have matching partition definitions, which is not my case. Which method should I use instead? I am trying to explore the docs of

MultiAssetSensorEvaluationContext

but I am kind of lost 😞