Hello all - I am looking to build a pipeline that...
# ask-community
b
Hello all - I am looking to build a pipeline that uses Airbyte to ingest data, and Dagster to orchestrate and transform the data. Each Airbyte connection corresponds to an integration that we have with one of our customers. I would love to be able to partition the data by each Airbyte connection in Dagster to: 1. Make the pipeline faster (when I update the data for one customer, I don't need all the data flowing through) 2. Make the pipeline more resilient (same issue as above, but preventing one customer data issues from affecting another. Is this possible with Dynamic Partitions? Seems like yes, but looking for some guidance. I have column called
orgId
which I want the value of to be the partition key