Alejandro A
03/06/2024, 8:44 PMScout
03/06/2024, 8:44 PMdynamic_partitioned_config
, the partition_fn
parameter is a function that generates a list of valid partition keys, which serve as the range of values that can be provided to the decorated run config function. This function can accept an optional datetime
argument representing the current time. The list of partitions returned by partition_fn
may change over time. Here is the relevant part of the documentation for `dynamic_partitioned_config`: python dagster.dynamic_partitioned_config(partition_fn, tags_for_partition_fn=None, tags_for_partition_key_fn=None)
- partition_fn ( Callable [ [ datetime.datetime ] , Sequence [ str ] ] ) – A function that generates a list of valid partition keys, which serve as the range of values that can be provided to the decorated run config function. The documentation does not explicitly state that you can pass additional parameters directly to partition_fn
through the dynamic_partitioned_config
decorator. The partition_fn
is expected to be a callable that takes an optional datetime
argument and returns a sequence of strings representing the partition keys. If you need to pass additional parameters to your partition_fn
, you would typically define those parameters within the scope of the partition_fn
itself or use a function that generates the partition_fn
with the parameters baked in. Here's an example of how you might define partition_fn
with additional parameters: python def get_values_partition(current_time, additional_param1, additional_param2): # Your logic here, using additional_param1 and additional_param2 pass def create_partition_fn(additional_param1, additional_param2): def partition_fn(current_time=None): return get_values_partition(current_time, additional_param1, additional_param2) return partition_fn my_partition_fn = create_partition_fn(my_param1, my_param2) @dynamic_partitioned_config(partition_fn=my_partition_fn) def my_partitioned_config(partition_key): # Your run config logic here pass
In this example, create_partition_fn
is a higher-order function that takes additional parameters and returns a partition_fn
that has access to these parameters. This allows you to customize the behavior of partition_fn
based on the additional parameters you provide.