https://dagster.io/ logo
#ask-ai
Title
# ask-ai
a

Alejandro A

03/06/2024, 8:44 PM
@Scout I have the following decorator to have a dynamic partition config, can I pass parameters to partition_fn? @dynamic_partitioned_config(partition_fn=get_values_partition)
s

Scout

03/06/2024, 8:44 PM
In the documentation for
dynamic_partitioned_config
, the
partition_fn
parameter is a function that generates a list of valid partition keys, which serve as the range of values that can be provided to the decorated run config function. This function can accept an optional
datetime
argument representing the current time. The list of partitions returned by
partition_fn
may change over time. Here is the relevant part of the documentation for `dynamic_partitioned_config`:
Copy code
python dagster.dynamic_partitioned_config(partition_fn, tags_for_partition_fn=None, tags_for_partition_key_fn=None)
- partition_fn ( Callable [ [ datetime.datetime ] , Sequence [ str ] ] ) – A function that generates a list of valid partition keys, which serve as the range of values that can be provided to the decorated run config function. The documentation does not explicitly state that you can pass additional parameters directly to
partition_fn
through the
dynamic_partitioned_config
decorator. The
partition_fn
is expected to be a callable that takes an optional
datetime
argument and returns a sequence of strings representing the partition keys. If you need to pass additional parameters to your
partition_fn
, you would typically define those parameters within the scope of the
partition_fn
itself or use a function that generates the
partition_fn
with the parameters baked in. Here's an example of how you might define
partition_fn
with additional parameters:
Copy code
python def get_values_partition(current_time, additional_param1, additional_param2):     # Your logic here, using additional_param1 and additional_param2     pass  def create_partition_fn(additional_param1, additional_param2):     def partition_fn(current_time=None):         return get_values_partition(current_time, additional_param1, additional_param2)     return partition_fn  my_partition_fn = create_partition_fn(my_param1, my_param2)  @dynamic_partitioned_config(partition_fn=my_partition_fn) def my_partitioned_config(partition_key):     # Your run config logic here     pass
In this example,
create_partition_fn
is a higher-order function that takes additional parameters and returns a
partition_fn
that has access to these parameters. This allows you to customize the behavior of
partition_fn
based on the additional parameters you provide.
2 Views