I want to create a partitioned job where the partition key i

Question

I want to create a partitioned job where the partition key is a primary key of a table in a database That way I can launch the job for each PK and run backfills I suppose this should use a `StaticPartitionsDefinition` but i don t understand how to set those in advance as the table grows

Rainer Pichler · Answer

Hello kyle As I understand it this would only be possible via a dynamic partitioned config But still in the supplied function you have to know the concrete partition values

kyle · Answer

So you are suggesting I set up the dynamic partitioned config partition fn to query my database for the set of primary keys

kyle · Answer

I got that working thanks for your input Although i have about 100k keys so it is a little slow to load the partitions What do you think of setting up the partition function to load the keys from an asset and update the asset a few times a day

kyle · Answer

As a separate scheduled job

Rainer Pichler · Answer

Could work Though in my use case I was able to derive the partition keys programmatically Note that I m a Dagster user and not a developer