Is there a new best practice for partitioning assets loaded from a dbt manifest in 0.15?
06/17/2022, 4:26 PM
hi @Alec Ryan there's no new guidance on partitioned dbt assets as of 0.15.0, although it is on our radar: https://github.com/dagster-io/dagster/issues/7683. For some extra background, in what way are you imagining partitioning your dbt assets? Would only incremental models have partitions, or would non-incremental models have them as well? Are you interested more in the backfilling capabilities of partitions, or something else?
06/17/2022, 5:38 PM
Yeah I imagine backfilling partitions would be helpful
I see a ton of value in incremental partitioning because they are partitioned by nature in some sense. Most cases I've seen use a date for example
06/22/2022, 6:44 AM
Hi @owen, right now our dbt model is using incremental strategy with start_time and end_time passed from dagster job. Specifically, we run partition ingestion job, daily. And when the daily ingestion finished, we pass the partition into dbt by variables to run dbt incrementally.