How good is the support for creating partitioned a...
# ask-community
r
How good is the support for creating partitioned assets that are not time-based? It seems most of the documented examples are for things like hourly, weekly, daily etc. If I wanted to create a partition that is based on something that is not time based, should I be worried about running into issues because partitions are assumed to be time-based? For some context, I would like to use something akin to customerID, where we are wanting to materialize specific assets for each customer. I don't want to use a static partition -- because we expect to add customers.
Is this sort of use-case intended for the Parition functionality?
Looking at the docs I am either trying to fit a round peg in a square hole -- or I am just thinking about partitions entirely wrong. Looking at
PartitionDefinition
it seems like time is a core aspect of a partition, except for static partitions.
z
Dagster definitely supports non-time partitioned assets / jobs. See static partitions and dynamic partitions. Though I would agree that the vast majority of docs seem to be geared toward time-based partitions and it'd be nice to see more examples, particularly for dynamic partitions. I'm currently using static partitions in a basic job to track a similar concept (credential_id for credentials that need to be rotated) I think there's just a lot more functionality implemented around time-based partitions because the problem and desired feature-set is more well-defined.
āž• 1
šŸ‘ 1
c
Agree that we could do a much better job of improving example documentation about the use cases for different partitions definitions. I think dynamic partitions are a good use case for you since you expect to add customers. We'll release documentation for this later today, but here's a link to the unreleased examples: https://docs.dagster.io/master/concepts/partitions-schedules-sensors/partitions#dynamically-partitioned-assets
šŸ‘ 2
r
Glad to see this on the roadmap! I think this will be a really killer feature. I will likely hold off on this for now and just rely on sensors/jobs to trigger a new run as needed.
šŸŒˆ 1
To provide some additional context for what I am trying to do here: ā€¢ We are an agtech company that scans crops and use AI to measure various aspects of the crop ā€¢ Each scan gets a scan-id associated with it ā€¢ I want to make scan-id a partition so that we can see all of the downstream data assets associated with a scan For now I think I can achieve most of what I want with sensors and RunRequests via
run_key
and
tags
.
s
@Ryan Ernst - to be clear, dynamic partitions are already in the latest release of Dagster (still marked experimental), so you can try them out whenever you're ready