Hi, I have a question on how to pass a dynamically created list to a static partition.
in the static partition example (continents), CONTINENTS is defined as a list up-front, but if I wanted to add a downstream asset to run partitions for all COUNTRIES in each CONTINENT, how would I approach that? (assume it's something like, for each continent, get list of countries and for each country, get the current population)
each of these partitions would be a different set of static partitions (I think), and assume for the sake of this example that the number of countries could change each time I need to run the job (not on a time schedule), so for each continent, I have to ping a DB to get the current list of COUNTRIES for that continent, then pass that list to another static partition which uses that list of countries as the partition_keys to Do the Things to get the population for that country.
is this possible (or advisable?) with static partitions? thanks!
06/16/2022, 4:52 PM
Hi John, this is an interesting question. In the past, we've considered implementing dynamic partitions on assets (allowing you to specify a function that returns the partitions for a given asset) which is something that we're still considering in the future but likely won't be implemented in the very near future.
Most of the software-defined asset focused views are dependent on having a defined and deterministic partition space, so we can display metrics like the number of partitions that have been materialized out of the total number of partitions. I think passing a variable list of partitions may result in some UI inconsistencies.
I think something you can consider is just defining your countries asset with a partitions definition containing the full list of countries, and then within the asset, pinging the DB to see if the current country partition is valid.
Another option is providing dynamic partitioned config, which allows you to specify a partition function that returns the set of partitions. You could consider using dynamic partitioned config with a regular job and yielding asset materialization objects within the job.
06/16/2022, 8:45 PM
Thanks Claire. I'll try the approaches you suggested (and probably ping you again if I run into an issue if that's ok).
The dynamic partitions feature you mention does sound like what I'm trying to do, so that gets a +1 from me 🙂 But I'm still not sure if I'm completely missing a fundamental concept or if my use case is really unique/distinct enough to warrant that feature.
Thanks again for the quick reply!