hi all, i'm trying to add dagster to an open-sourc...
# announcements
hi all, i'm trying to add dagster to an open-source project i'm working on, got stuck, and thought i'd come here for some guidance. basic issue is i need to call an API (Socrata) with each of a list of keys and merge the results...
i have a solid that does the download and works great when i configure the key as an input in YAML. i'm hoping to keep this as a standalone solid and then reference it once for each dataset
i tried to do this from the pipeline, but that doesn't seem like the right approach. next i tried creating a composite solid and reading the keys from YAML and aliasing the download solid for each key, but it looks like composite solids don't support context and therefore i'm not sure how to configure it
i see there is a mechanism for creating configurations on the fly which i haven't tried yet
it also seems like another possibility would be running a pipeline for each key and having another pipeline run after to do whatever merging/cleaning needed to happen
anyway, before i go too far down any other of these other paths i thought i'd see if there was any guidance from the group here...w
what's the best way to dynamically run the same solid from a list (ideally within the same pipeline)?
thanks in advance!
Hey @Matt Webster, is your list of keys fixed for each pipeline?
🙌 1
If so, you might find this example helpful: https://docs.dagster.io/examples/fan_in_pipeline
yep. it's the same keys. i'm looking to do a map-reduce style ETL process. i missed this example in the docs. will take a look and give it a shot. thanks @sashank!
👍 1
Lmk if that ends up not being what you were looking for