AJ Floersch
02/21/2023, 1:45 PM@asset
def render_asset_1(context):
# ... do something...
return something
assets = [render_asset_1, ...]
@op
def determine_assets_to_launch(context):
for asset in assets:
# ... some logic on which assets are currently needed...
if criteria:
asset() # Run the asset
@job
def dynamic_asset_job():
# the job that will ultimately run on a schedule.
determine_assets_to_launch()
My reason for something like this is I would like to have a series of explicitly defined assets that can take full advantage of dagit and adhoc materializations, while also being able to dynamically execute the assets through a single job. Something that doesn't seem possible with just define_asset_job
based on an AssetSelection
group which will always execute every Asset, nor can it be done through something like just an op with AssetMaterialization
, which doesn't render the Asset in dagit for adhoc launches.Saul Burgos
02/21/2023, 2:54 PMAJ Floersch
02/21/2023, 2:59 PMSaul Burgos
02/21/2023, 2:59 PMAJ Floersch
02/21/2023, 3:03 PMcontext.instance.get...()
, but how would I do something like this outside of that?Saul Burgos
02/21/2023, 3:56 PMclaire
02/22/2023, 6:48 PMwith DagsterInstance.get() as instance:
...