https://dagster.io/ logo
#ask-community
Title
# ask-community
h

Harry Park

06/28/2023, 11:29 AM
Question on materializing assets. I’m migrating an existing C# pipeline. I’ve made that work using Jobs and Ops with dagster-shell. I’m at SourceAssets now. I want to use dagster’s to help see gaps in the the date partition keys. 1) How do I get externally created assets (files) into Dagster’s asset system? Scheduled job for a Sensor op to emit AssetObservations of SourceAssets? 2) Do I then write an IO manager to load and write these assets? Or is this optional. 3) If I want to run backfills on the SourceAsset, do I make an Op that calls the execute_shell_script C# program with the keys to make the asset and then have it emit an AssetObservation?
🤖 1
s

sean

06/28/2023, 2:25 PM
1) How do I get externally created assets (files) into Dagster’s asset system? Scheduled job for a Sensor op to emit AssetObservations of SourceAssets?
Yes, model them as
SourceAssets
. One approach is to use observable source assets to generate
AssetObservations
.
2) Do I then write an IO manager to load and write these assets? Or is this optional.
If they are specified as dependencies in the usual way, then your IO manager will need to be able load them (not write them if they are source assets). Or you can use
non_argument_deps
and write whatever logic you want to load them inside of an upstream asset compute function (i.e. skip the IO manager)
3) If I want to run backfills on the SourceAsset, do I make an Op that calls the execute_shell_script C# program with the keys to make the asset and then have it emit an AssetObservation?
“Backfills of a SourceAsset” is an undefined notion in Dagster, because a
SourceAsset
is by definition an asset for which Dagster does not control materializations.
h

Harry Park

06/28/2023, 4:33 PM
Ahh, that 3rd point is great. so if i do want to do backfills, i need to make it a regular asset. Thanks, I see now why assets are so important to dagster.
b

Ben Ogorek

07/01/2023, 2:19 AM
Curious, are there any examples where a dagster shell op is used to define a graph-based asset? I.e., make it a regular asset
s

sean

07/01/2023, 6:04 PM
Hi Ben, I’m not aware of any examples in our code but this works:
Copy code
from dagster import graph_asset
from dagster_shell import create_shell_command_op

hey_op = create_shell_command_op("echo hey", name="hey_op")

@graph_asset
def hey():
    return hey_op()
❤️ 1
b

Ben Ogorek

07/01/2023, 9:26 PM
Thanks @sean that will get me started! I think my next goal will be to “register” a file that one of those shell ops creates as an asset.
h

Harry Park

07/01/2023, 9:27 PM
@Ben Ogorek let me know if you have any success. I want to do similar things
b

Ben Ogorek

07/01/2023, 9:30 PM
Will do @Harry Park. Glad to know someone else is interested
2 Views