Question on materializing assets. I’m migrating an existing C# pipeline. I’ve made that work using Jobs and Ops with dagster-shell. I’m at SourceAssets now. I want to use dagster’s to help see gaps in the the date partition keys. 1) How do I get externally created assets (files) into Dagster’s asset system? Scheduled job for a Sensor op to emit AssetObservations of SourceAssets? 2) Do I then write an IO manager to load and write these assets? Or is this optional. 3) If I want to run backfills on the SourceAsset, do I make an Op that calls the execute_shell_script C# program with the keys to make the asset and then have it emit an AssetObservation?

1) How do I get externally created assets (files) into Dagster’s asset system? Scheduled job for a Sensor op to emit AssetObservations of SourceAssets?

Yes, model them as

SourceAssets

. One approach is to use observable source assets to generate

AssetObservations

.

2) Do I then write an IO manager to load and write these assets? Or is this optional.

If they are specified as dependencies in the usual way, then your IO manager will need to be able load them (not write them if they are source assets). Or you can use

non_argument_deps

and write whatever logic you want to load them inside of an upstream asset compute function (i.e. skip the IO manager)

3) If I want to run backfills on the SourceAsset, do I make an Op that calls the execute_shell_script C# program with the keys to make the asset and then have it emit an AssetObservation?

“Backfills of a SourceAsset” is an undefined notion in Dagster, because a

SourceAsset

is by definition an asset for which Dagster does not control materializations.

Ahh, that 3rd point is great. so if i do want to do backfills, i need to make it a regular asset. Thanks, I see now why assets are so important to dagster.

Curious, are there any examples where a dagster shell op is used to define a graph-based asset? I.e., make it a regular asset

Hi Ben, I’m not aware of any examples in our code but this works:

from dagster import graph_asset
from dagster_shell import create_shell_command_op

hey_op = create_shell_command_op("echo hey", name="hey_op")

@graph_asset
def hey():
    return hey_op()

Thanks @sean that will get me started! I think my next goal will be to “register” a file that one of those shell ops creates as an asset.

@Ben Ogorek let me know if you have any success. I want to do similar things

Will do @Harry Park. Glad to know someone else is interested