Hello - what is the idiomatic way for dagster to make requests to an api - and job won't complete for a while - and you can download later. I can see where we can have a scheduled asset to make the requests - but does dagster allow for fine control of asset retries (like try for a week once an hour) - or my idea was to split the task into two asset pipelines - one to make the request - and the other to check on pending requests stored in a db table. While that's possible - it doesn't seem like an idiomatic way to solve the problem in dagster. Dagster should have the tools for an asset in an asset graph to poll until complete with fine grain control and then complete the asset graph when the asset is able to be materialized.

I’m new to Dagster, but concepts and terminology are similar to others tools, mainly Airflow. You may want to read about sensors, which may or may not work similarly--I have no direct experience with them in Dagster. There are still open questions about your architecture, such as whether or not the remote service can call back, etc. If it could you could not “retry or check regularly” and instead let it simply trigger a new job asynchronously. Again, just thinking out loud, but sensors would be the first thing I’d look into.

I think the RetryRequested may be the best option for me at this point. Thanks for the input though.

hi @Matthew Karas! I think setting a RetryPolicy on your asset would be your best bet here -- it's similar to just raising RetryRequeted, but a bit more declarative

Is there a way to retry forever?

there's no explicit way of doing this, but you could just set max_retries to a massive number