Hi everyone! I had a question regarding dynamically generating assets. Lets say I have a python function that loops through a list which triggers it to read 10 from different API tables. How can I dynamically generate Dagster Assets for all 10 tables (maybe in a for loop?) without having to manually define the names of each asset in an @asset decorator?

Hi Matt, Do you know the names of the tables in advance, or do you need to determine this at runtime. If the former, it sounds like the best solution for you might be a partitioned asset. This requires just one

@asset

definition. If the latter, then I’ll call in a colleague who knows more about the status of dynamic partitions (active area of development for us right now).

Hey Sean, thanks for looking into this. Yes I do know the table names in advance, they would be in the list. But each list item represents a different table & has a different schema, so I dont think I could just use a partition by date if Im understanding the docs correctly (assuming this is the same concept as a database table partition)? I think the problem is in my case is that the asset name comes from the function name. So maybe it would be along the lines of dynamically creating functions? So for example, here would be the code (in theory, I know this wouldnt work):

list_of_tables = ['table name 1','table name 2','table name 3']

for tbl in list_of_tables:
    @asset 
    def dynamic_asset_name_here():
        df = function_that_gets_api_data_to_dataframe(tbl)
        return df

Dagster partitions are more like arbitrary axes than traditional database partititons-- there is no reason a partition needs to correspond to time/date, it can represent anything-- so it still might be appropriate for you. If you do for whatever reason find you need to dynamically create separate assets though, you can override the function name for the name of the asset

assets = []
for tbl in list_of_tables:
    @asset(name=<something_dynamic>)
    def dynamically_named_asset():
        ...
    assets.append(dynamically_named_asset)

I recently wrote a short guide that might be helpful here: https://github.com/dagster-io/dagster/discussions/12061