I am curious if assets can help in the following case. I have a job with the following ops: • t1 = create table1 if not exists • t2 = create table2 if not exists • m1 = create materialized view based on table1 and table2 if not exists • f1 = fill table1(t1) • f2 = fill table(t2) • refresh the materialized view (m1, f1, f2) All the ops at this point write directly to the database, but conceivably I could extract that functionality out. Is there a way to do this with assets? The main things I am looking to gain are: • an easy way for consumers to tell when a table was last updated • Ideally some metadata about the updates • Some way to not forget to refresh the materialized view when the upstream data gets updated

hi daniel, the way I would conceptualize this as a group of assets is: • Asset Table1: Do t1 + f1 • Asset Table2: Do t2 + f2 • Asset View1: Do m1 + refresh, depends on Table1 and Table2

I am a bit confused on the documentation. Should I be using `@asset`or

@op

like I currently am and then in the op doing the actual work?

you should replace your

@op

decorators with

@asset

(but the functions underneath the decorators would remain basically the same, other than combining together the bodies of a couple of the ops)

got it. Can assets call

@op

or no?

and I think you can skip using a custom io_manager (my guess is that it would be more convenient to just include the metadata-generating code in the op body)

is there a way to differentiate between asset creation and asset "update"?

under the hood,

@asset

will create an op for you (and will do some other stuff to help dagster figure out how to connect the generated ops to other ops), so the code underneath the asset decorator has the same constraints as code underneath the op decatorator (so no, you shouldn't create ops manually in those functions)

perfect, I'm going to give it a go

no problem 🙂

how do I pass an asset as non_argument_deps? I see that the param exists in the docs, but not how to use it

oops that's an oversight. the argument's type is just a set of AssetKeys. So you pass in the set of AssetKeys that you want this asset to depend on (in your case it should be something like

non_argument_deps={AssetKey("table1"), AssetKey("table2")}

)

and then dagster figures out the graph for all of those automagically? (I currently pass a bunch of

Nothing

around to make it it all work)

yep exactly 🙂 -- you can toss the assets into an AssetGroup, and dagster will hook up all the op inputs/outputs in a way that satisfies the asset dependencies that are defined

🙂

for the first question, once you have an AssetGroup, you can call .build_job() on it to get a job that will update all those assets (and this job can then be put on a schedule just like any other job)

I think what I was trying to do might be supported by partitioned assets

hm interesting -- can you say more?

basically, I had an

op

that took in a list of files to process as part of the config

with these files, do you combine their contents together into one larger thing in the op, or do you basically do some sequence of actions for each (deduped) file, with these actions spanning multiple ops?

Not 100% sure what the difference is. Imagine 10 files, with 100 records each. Some records might appear in multiple files. Process each record in each file, and write them to a single table (order matters for clobbering)

could it make sense to model that list of files as an asset itself?

not logically. I understand that it might work to get the data passed to an asset though. Do partitions only really support time?

partitions definitely support non-time dimensions, so your original suggestion seems like it should work in theory, the tricky part is just that you can generally only launch a job for a single partition at a time. If you model partitions as 1 file = 1 partition, then you run into issues. You could hack around the immediate issue by swapping out op config with resource config, and reading the configuration of the resource inside the asset to determine what files to read, but this might be a bit of a bad rabbit hole to go down, as it doesn't really mesh well with the asset model, which sort of assumes that you're able to just execute the asset in a vacuum (without need for more configuration)

h'm

the most "asset-y" way of doing this would be to include the code to generate the files that you need as part of the asset that combines them together, but I recognize that might not be ideal depending on your exact setup

the core issue with that is that it is hard to call ops from other ops

if you could create an asset from an @graph (rather than just an @op), would this be sufficient? this is functionality that we are planning to include in the nearish future

I think so, because then I can effectively call an

op

from an

op

great

now I'm getting an

AssetsDefinition object has no attribute 'asset_key'.

Function decoration looks like:

@asset(ins={'files': list_of_files_asset}, required_resource_keys={'r1','r2'}, non_argument_deps={AssetKey('<name of upstream asset')})

oh weird -- can I see more of the stack trace?

it just goes to

run_code

I just figured out

@asset(ins={'foo': AssetIn(AssetKey('my_asset_key'))})