When testing software defined assets that make use of time r dagster #ask-community

When testing software defined assets that make use...

Brendan Jackson

06/28/2023, 9:23 AM

When testing software defined assets that make use of time range partitions, what's the best way to assemble the context?

build_op_context

allows a single partition key, not a range

🤖 1

Brendan Jackson

06/28/2023, 9:44 AM

Looks like this is related: https://github.com/dagster-io/dagster/pull/13088

sean

06/28/2023, 1:16 PM

Hi Brendan, It looks like we don’t support this right now-- partition key ranges are a bit rough around the edges. The technical reason is that these ranges are sourced from run tags, and there are no run tags when doing direct op invocation. However, I think it’d be simple enough to add support for this to

build_op_context

, if you create a issue for it I’m pretty sure I can knock it out quickly.

Brendan Jackson

06/28/2023, 1:38 PM

I @sean - that's interesting. I will make an issue, but I'd like a bit of clarity first, as I actually got it to work with:

Copy code

context = dagster.build_op_context(partition_key="2023-01-05")

and then

time_window = context.asset_partitions_time_window_for_output()

works. Obviously this doesn't cover a range beyond a single day (which I would like to test). (I notice that this method is marked

deprecated

on the

OpExecutionContext

(but not the

AssetExecutionContext

) - which leaves me a bit confused. What's going on here?) Finally, this

context.partition_time_window

does not work, despite the above working. This is another confusing part! Any ideas before I make the issue?

Brendan Jackson

06/28/2023, 1:44 PM

https://github.com/dagster-io/dagster/issues/14990

sean

06/28/2023, 1:51 PM

If we have a daily partitioned asset, then each key corresponds to a timespan over a whole day (a “time window”)-- so, no surprise that

asset_partitions_time_window_for_output

works, since it can be derived from one partition key.

(I notice that this method is marked deprecated on the OpExecutionContext (but not the AssetExecutionContext) - which leaves me a bit confused. What’s going on here?)

I’m not 100% up to date here but that’s probably bc

OpExecutionContext

is only being used now in situations where there is no associated asset. If you have an

@asset

it should always receive the

AssetExecutionContext

subclass instead.

Finally, this context.partition_time_window does not work, despite the above working. This is another confusing part! Any ideas before I make the issue?

Seems odd, what error gets thrown?

👍 1

Brendan Jackson

06/28/2023, 1:52 PM

Copy code

AttributeError: 'BoundOpExecutionContext' object has no attribute '_step_execution_context'. Did you mean: 'get_step_execution_context'?

Brendan Jackson

06/28/2023, 1:53 PM

I'll add the error to the issue

sean

06/28/2023, 2:07 PM

Thanks, seems like a straightforward bug

🙏 1

Brendan Jackson

06/28/2023, 2:10 PM

With the bug fixed, presumably inside an

@asset

context.partition_time_window

should be preferred to

context.asset_partitions_time_window_for_output()

? The latter I'd expect from the name to be more associated with output steps.

sean

06/28/2023, 2:32 PM

I agree

Brendan Jackson

08/04/2023, 10:22 AM

Hi Sean - wondering if you managed to track down the bug regarding

asset_partitions_time_window_for_output()

partition_time_window

? The former works in tests, the latter does not. Latest version of Dagster I also still can't use ranges (using

PartitionKeyRange

when building a context results in

Tried to access partition_key for a non-partitioned asset

, while

partition_key

alone does not).

Brendan Jackson

08/04/2023, 10:33 AM

Copy code

>       return self._step_execution_context.partition_time_window
E       AttributeError: 'BoundOpExecutionContext' object has no attribute '_step_execution_context'. Did you mean: 'get_step_execution_context'?

4 Views

Open in Slack

Previous Next