https://dagster.io/ logo
#ask-community
Title
# ask-community
s

Spencer Nelson

02/23/2023, 1:22 AM
How do I add timeouts to asset materializations?
c

claire

02/23/2023, 5:11 PM
Hi Spencer. What's the behavior you're looking for, do you mean timeouts for the asset execution?
s

Spencer Nelson

02/23/2023, 5:24 PM
Sorry, yes - I mean for asset execution
In particular, I am trying to build a partitioned asset by querying an extremely flakey database with an obscure interface (Virtual Observatory Table Access Protocol). The client library itself doesn’t support timeouts, and adding them to the client isn’t really feasible. About 25% of queries hang indefinitely, probably because of a bug on the remote end, which I don’t control. Usually, my queries complete in about 1-5 minutes, but I have some that have been running for about 16 hours and will probably never complete.
I have about 2,500 partitions of this type to build. Right now, the only thing I am able to do is kick off runs, check back in an hour, manually terminate the ones that are still going, and repeat. This will take a very long time to do.
I’d like to be able to say
@asset(max_execution_time=datetime.timedelta(minutes=20))
. I’d like the asset execution to be aborted and retried (with the retry policy I passed in) if execution passes that threshold.
Let me know if you’d prefer I write this up in a Github issue, by the way
c

claire

02/23/2023, 6:56 PM
Yep, the use case makes a lot of sense. I think this is something we'd like to support, but we haven't gotten around to. Here's a long-lived github issue that you can track: https://github.com/dagster-io/dagster/issues/3666
👍 1
2 Views