https://dagster.io/ logo
#ask-community
Title
# ask-community
y

Yepher

08/26/2023, 3:00 PM
I am learning Dagster, so sorry if this is a very basic question. Here is what I am trying to understand in Assets. I have a call to a database or webserver that returns an array of values lets call them customer IDs. For each cutomer ID I want to run an asset to analyze that customer. Here is a pseudo code of the idea:
Copy code
# Find array of customer_ids to analyze
@asset
def find_customers_to_analyze(context):
    with database.get_resource().get_client() as client:
        detector = CustomerWrapper(context)
        return detector.find_customrs_to_analyze() # returns an array of zero ore more customer_ids


# For each customer_id returned by find_customers_to_analyze
@asset
def analyze_customer(context, customer_id):
    with database.get_resource().get_client() as client:
        detector = CustomerWrapper(context)
        return detector.check_for_anomalies(customer_id)
It would be nice if
analyze_customer
would run as many as possible in parallel as allowed by some concurrent limit. Is there some documentation or tutorial somewhere that describes how to achieve this. I assume this is a very common usecase but my Google-foo is not strong enough to find what I am looking for. I found similar unanswered question os SO
m

martin o leary

08/26/2023, 3:39 PM
y

Yepher

08/26/2023, 4:38 PM
Thanks @martin o leary that document uses
ops
Can I do the same thing with assets or must I use ops?
Maybe this? https://dagster.io/blog/dynamic-partitioning `
Copy code
DynamicPartitionsDefinition
?
m

martin o leary

08/26/2023, 5:20 PM
I think of partitions as being “outside the asset” if that makes sense? You could build a dynamic graph as above and create an asset from it maybe? https://docs.dagster.io/concepts/assets/graph-backed-assets#graph-backed-assets
👍 1
2 Views