https://dagster.io/ logo
#ask-community
Title
# ask-community
d

Drew You

03/30/2023, 2:02 PM
I have an asset that defines a universe of data that another asset wants to create a time series from. Ideally, I would be able to only trigger runs for the diff of the universe (i.e. only things that were added to it, so a strict dependency doesn't seem to work because this would refresh the downstream asset entirely each time there were a change to the universe) and the missing parts of the time series that I need. I am trying to use a
Multipartition(TimeWindowPartition,DynamicPartition)
but that doesn't work because
MultiPartition
doesn't support
DynamicPartition
. What would be the best approach to this?
Copy code
@sensor(job=pair_data, required_resource_keys={"duckdb"})
# or should this be a job?
def pairs_partition(context):
    current_instruments = context.resources.duckdb.query(
pair_data.to_source_asset()))
    partition_instruments = instruments_partition.get_partitions(
        context.instance)
    added_instruments = current_instruments - partition_instruments
    removed_instruments = partition_instruments - current_instruments
    context.instance.add_dynamic_partitions(
        instruments_partition.name, added_instruments)
    context.instance.remove_dynamic_partitions(
        instruments_partition.name, removed_instruments)
    return my_job.run_request_for_partition(partition_key, instance=context.instance)
Here's the stab at how I wanted to use the
DynamicPartition
if that makes it more clear.
c

claire

03/30/2023, 6:04 PM
Hi Drew. I responded on the github issue, but the gist of why you're seeing this error is because the following things happened in order: • the asset had a single-dimension partitions def and some partition(s) failed • you converted the asset to a multipartitions def • dagster tried to convert the previously failed partitions into partition keys of the current multipartitions def I have a PR out to fix the validation issue: https://github.com/dagster-io/dagster/pull/13268 Dynamic dimensions in multipartitions defs should still work, in the meantime you can wipe that asset or switch to using a new asset key.