Stephen Bailey
03/13/2024, 1:39 PMScout
03/13/2024, 1:39 PMpython from dagster import job, op, static_partitioned_config CONTINENTS = [ "Africa", "Antarctica", "Asia", "Europe", "North America", "Oceania", "South America", ] @static_partitioned_config(partition_keys=CONTINENTS) def continent_config(partition_key: str): return {"ops": {"continent_op": {"config": {"continent_name": partition_key}}}} @op def continent_op(context, continent_name: str): <http://context.log.info|context.log.info>(f"Processing data for {continent_name}") @job(config=continent_config) def continent_job(): continent_op()
In this example, the job continent_job
is partitioned by continents. Each partition corresponds to a continent. The continent_config
function generates a configuration for each partition, which is used to pass the partition key (in this case, the name of a continent) to the continent_op
op. The continent_op
then logs a message indicating it is processing data for the specified continent.