I m having a tough time understanding the relationship or la

Question

I m having a tough time understanding the relationship or lack thereof between pipeline presets and partitions For example I have an existing pipeline with some preset defs for `local` and `remote` configs Now I want to add partitions to this The `PartitionSetDefinition` takes a `run config fn for partition` function that returns a config But there s no way to know what the current config is in the function Whatever I return complete replaces the preset configs In Dagit the dropdown in the playground shows presets and partition sets as mutually exclusive selections I was anticipating being able to select a preset and then which partition to operate on Therefore the pipeline config would be a preset config merged with a partition config Is this possible to do

prha · Answer

Both presets and partition sets are just convenience wrappers around config generation

prha · Answer

Presets are hard coded config while partitions are more dynamic and are based off of some logical partitioning

prha · Answer

To the extent that both generate full pipeline config there s not a clear way to resolve the intersection of them

prha · Answer

I think other folks have manually done the override within the `run config fn for partition` by loading the config from the static preset they want to use and then overriding certain fields < sashank> might be able to confirm

prha · Answer

In your example we would potentially need to set up two different partition sets corresponding to local and remote execution e g `catalog partitioned local` and `catalog partitioned remote`

King Chung Huang · Answer

Hmm thanks for the info That s rather unfortunate This is a simple example but I can see more complex ones exploding in Partition Sets for every permutation of preset partition

prha · Answer

Would love to hear what your ideal behavior would be using presets partitions together

prha · Answer

We erred on the side of having Dagster not have to resolve any configuration conflict but it s useful to hear how you re thinking about it

King Chung Huang · Answer

I had in my mind how Docker Compose and GitLab CI configs work Multiple configs that are deep merged together So I d like to be able to select the `local` preset and then have the `PartitionSetDefinition` only return the specific partition config which Dagster would merge into the config

King Chung Huang · Answer

I ll write up a feature request on GitHub

King Chung Huang · Answer

Created proposal for extending partition set configs from preset configs <https github com dagster io dagster issues 2704>