https://dagster.io/ logo
Title
a

AJ Floersch

02/21/2023, 1:45 PM
Hoping I'm just missing something in the docs, but I'm struggling to figure out how to launch Asset materializations from an op or job, at least in a way that allows me to really customize which Assets get launched at runtime. Please excuse the rough snippet below, but something along the lines of...
@asset
def render_asset_1(context):
    # ... do something...
    return something

assets = [render_asset_1, ...]

@op
def determine_assets_to_launch(context):
    for asset in assets:
        # ... some logic on which assets are currently needed...
        if criteria:
            asset() # Run the asset


@job
def dynamic_asset_job():
    # the job that will ultimately run on a schedule.
    determine_assets_to_launch()
My reason for something like this is I would like to have a series of explicitly defined assets that can take full advantage of dagit and adhoc materializations, while also being able to dynamically execute the assets through a single job. Something that doesn't seem possible with just
define_asset_job
based on an
AssetSelection
group which will always execute every Asset, nor can it be done through something like just an op with
AssetMaterialization
, which doesn't render the Asset in dagit for adhoc launches.
s

Saul Burgos

02/21/2023, 2:54 PM
Check this link
Basically you need to create a assetSelection based on the operation according to the doc like: + - | and use that selection when you run your job
a

AJ Floersch

02/21/2023, 2:59 PM
Gotcha! And so then the AssetSelection will be evaluated prior to each job run to determine which Assets it will be running that time?
s

Saul Burgos

02/21/2023, 2:59 PM
Here is the doc, is very limited but is what we have: https://docs.dagster.io/_apidocs/assets#dagster.AssetSelection
The only way I could do this was in this way: I create the job previously and later I execute it inside a operation using "execute_in_progress" that allow you to pass "asset_selection"
a

AJ Floersch

02/21/2023, 3:03 PM
Ah okay! Last question - in a scenario like this, is there a way to get hold of basic Asset event details such as last materialization time? I know how to do so within an op/asset using
context.instance.get...()
, but how would I do something like this outside of that?
Basically would like to use some of that info in determining my _selected_asset_keys._
s

Saul Burgos

02/21/2023, 3:56 PM
If I understand well, you can use a sensor and get all the that information
c

claire

02/22/2023, 6:48 PM
In response to the last question about getting basic asset event details, it's easiest to do it inside of an op/asset/sensor where you have access to the instance and can fetch the latest materialization per asset, etc. Outside of that, you'd probably have to get your dagster instance and then fetch the information you need.
with DagsterInstance.get() as instance:
   ...