https://dagster.io/ logo
#ask-community
Title
# ask-community
t

Todd

09/05/2022, 7:32 PM
Hi all. I could just be trying to use assets incorrectly but I have an asset that is the result of an http call. I need to do something computationally expensive with that asset. I am using dynamic outputs to do the computation efficiently. I then return a call to collect() from the result of a map() in my graph but that doesnt seem to be allowed? I need to somehow convert the fanin into a list or op output prior to returning? My plan was to make this a "graph-backed asset".
🤖 1
j

Jon Simpson

09/05/2022, 7:40 PM
Collect() is a function called in a Graph/Job not inside an Op. So
Copy code
@op(out=DynamicOut())
def first():
  for i in range(10):
    yield i

@op
def fetch_a_lot_of_data(id: int):
  results : List[int] = fetch_data(id)
  return results

@op
def sum_stuff(large_list:List[int]):
  return sum(large_list)

@graph
def my_job():
  mystuff = first()
  results = mystuff.map(fetch_a_lot_of_data)
  summed = sum_stuff(results.collect())

#do asset from graph stuff
t

Todd

09/05/2022, 7:42 PM
I am calling it from inside the graph and returning it: return thing().map(expensive_thing).collect()
but it doesn't let me return the fanin
this works:
Copy code
return fake_op_that_does_nothing(thing().map(expensive_thing).collect())
i bet i'm not understanding something fundamental
For anyone else that is having this problem, it was because my @graph had a different name than the AssetDefinition.from_graph variable.
5 Views