https://dagster.io/ logo
#ask-community
Title
# ask-community
m

Malo PARIS

03/31/2023, 9:18 AM
Hi all, need some help 🙂 I try to parametrize all similar assets based on Yaml file. How to catch asset metadata (name, keys, ...) during an asset execution ? Example of what I try to :
Copy code
@asset()
def test_asset(context):
    print(context.asset_name)
    >>> test_asset
Thanks !
dagster bot responded by community 1
🤖 1
v

Vinnie

03/31/2023, 10:11 AM
This should be possible. You can check everything that’s passed to the
context
arg by inspecting the
OpExecutionContext
Class: https://github.com/dagster-io/dagster/blob/master/python_modules/dagster/dagster/_core/execution/context/compute.py#L91 From then it’s just a matter of figuring out what information you need exactly
❤️ 1
m

Malo PARIS

03/31/2023, 11:12 AM
I will look at it, thanks 🙂
v

Vinnie

03/31/2023, 11:13 AM
I would maybe look into what you’re trying to do exactly though, in many cases, using partitioned assets might be the way to go.
m

Malo PARIS

03/31/2023, 11:21 AM
Today, I create one extraction asset for each source table. The format looks always the same, only the parameters changes. Example :
Copy code
@asset(key_prefix=["ETL"],required_resource_keys={"my_sap_co"},retry_policy=sap_retry_policy)
def SAP_TABLE1(context):
    
    my_connection = context.resources.my_sap_co
    SAP_TABLE1 = my_connection.get_table(
      table="SAP_TABLE1",
      fields="F1,F2,F3",
      filters=["Filter1","Filter2"])
  

    return SAP_TABLE1
I would like to create a Yaml file containing all the parameters for each asset instead. Example :
Copy code
asset:
  SAP_TABLE1:
      fields: F1,F2,F3
      filters: 
        - ["Filter1","Filter2"]
  SAP_TABLE2:
      fields: ....
      filters: 
        - [....]
The ultimate goal is to be able to create assets on the fly by reading this Yaml file :)
c

Charles

03/31/2023, 11:42 AM
What I'm trying do is very similar to this, if you find an example or have one you can share that'd be much appreciated!
v

Vinnie

03/31/2023, 11:46 AM
Right, in case you wanna track these assets separately (read: not as partitions of a single asset) I’d probably use an asset factory type of logic, something like:
Copy code
def make_sap_table(conf):
    my_assets = []
    for item in conf:
        @asset(name=item.name, key_prefix=["ETL"],required_resource_keys={"my_sap_co"},retry_policy=sap_retry_policy)
        def table_asset(context):
            
            my_connection = context.resources.my_sap_co
            SAP_TABLE = my_connection.get_table(
            table=item.table,
            fields=item.fields,
            filters=item.filters)
        

            return SAP_TABLE
        my_assets.append(table_asset)

    return my_assets
❤️ 1
m

Malo PARIS

03/31/2023, 11:49 AM
very cool ! I will try it 🙂 thanks
2 Views