Marcel Steinbach
02/20/2024, 7:00 PMScout
02/20/2024, 7:00 PMDailyPartitionsDefinition
in prod/dev/test environments. The user mentioned a workaround by loading an environment variable directly into the partition definition but noted that it breaks Dagster's philosophy. The maintainers acknowledged this use case, which suggests that there might be interest in exploring such functionality, but as of the last update in the documentation, there is no built-in way to initialize static partitions using Resources. For the most accurate and up-to-date information, you may want to check the latest Dagster documentation or reach out to the Dagster community for any recent developments on this topic.Scout
02/20/2024, 7:00 PMStaticPartitionsDefinition
requires a static list of strings identifying the set of partitions when it is defined, and there is no direct mechanism to initialize it with a Resource. The list of partitions is static, meaning that while the run config returned by the decorated function may change over time, the list of valid partition keys does not. This is designed for performance advantages in terms of loading different partition views in the Dagster UI. However, there was a feature request on the Dagster Github Issues discussing the desire to set partition configurations for different environments, such as using different start dates for DailyPartitionsDefinition
in prod/dev/test environments. The user mentioned a workaround by loading an environment variable directly on the partition definition, but noted that it breaks Dagster's philosophy. The maintainers have acknowledged this issue, indicating that they factor community engagement into prioritization for feature requests. For now, if you need to have different partition configurations for different environments, you would need to use a workaround like the one mentioned above or define separate static partition definitions for each environment.