Christian Rønsholt
02/26/2024, 12:41 PMScout
02/26/2024, 12:41 PM@daily_partitioned_config
decorator to define the partitioned config for your asset. 2. Create a schedule from the partitioned job: Use the build_schedule_from_partitioned_job
function to create a schedule for your partitioned asset job. This function will automatically handle the partition keys for each scheduled run based on the partition definition. 3. Test the schedule: To test the schedule, you can use the execute_in_process
method with a specific partition_key
to simulate the execution of the asset for a given partition. For example: python assert my_daily_partitioned_asset_job.execute_in_process(partition_key="2020-01-01").success
Regarding the config in the launchpad, when you select a specific partition in the Dagster UI Launchpad, the run config for that partition should be automatically populated in the editor. The partition key is used to determine the configuration for that specific run based on the partitioned asset's definition. The documentation provides an example of how the run config might look when a partition is selected in the UI Launchpad: python @daily_partitioned_config(start_date=datetime(2020, 1, 1)) def my_partitioned_config(start: datetime, _end: datetime): return { "ops": { "process_data_for_date": {"config": {"date": start.strftime("%Y-%m-%d")}} } }
In the UI, when you select a partition (e.g., 2020-01-02
), the corresponding run config should reflect the configuration for that date, with the date
field populated with the partition key. Remember that the actual configuration will depend on how you've defined your partitioned asset and the logic within your partitioned config function. The example provided is a general guide, and you should adjust it according to your specific use case and the structure of your Dagster project.