Julius
04/03/2023, 8:39 AMdynamic partition
for a CDC load (an dbt incremental materialization).
How can i “initialize” the partition have 1 partition named INITIAL_LOAD
and then use a sensor to add further paritions ?
Also, how can I record a data against this partition as my start date ?
fivetran_prod_test_connection_test_03 = DynamicPartitionsDefinition(
# <https://docs.dagster.io/_apidocs/partitions#dagster.DynamicPartitionsDefinition>
name="fivetran_prod_test_connection_test_03",
)
fivetran_prod_test_connection_test_03.instance.add_dynamic_partitions(fivetran_prod_test_connection_test_03.name, ['INITIAL_LOAD'])
# Where can I add a start_date config to this partition ?
Another question:
Can we specify the name of dbt ops that it created using the dbt asset? Like in this example, the dbt run name "run_dbt_3a4be". Can we config this name?sandy
04/03/2023, 7:55 PMHow can i “initialize” the partition have 1 partition namedIn a script, you could do this following:and then use a sensor to add further paritions ?INITIAL_LOAD
instance = DagsterInstance.get()
instance.add_dynamic_partitions("partitions_def_name", ["INITIAL_LOAD"])
Also, how can I record a data against this partition as my start date ?Are you able to explain what you mean by this in more detail? There isn't currently the capability to attach metadata to a partition.
Can we specify the name of dbt ops that it created using the dbt asset? Like in this example, the dbt run name "run_dbt_3a4be". Can we config this name? (edited)This isn't currently possible. If you file a Github issue for this, we might be able to add support for it.