@Scout Is there a way to keep track of an increment value in a op so when i fail it can restart from that point?

In Dagster, if you want to keep track of an increment value within an op and ensure that upon failure the op can restart from that point, you would typically handle this through the use of retries and possibly by persisting state externally. However, the documentation does not provide a direct example of maintaining an increment value within an op that persists across retries. For retries, you can configure a retry policy on your op that specifies the maximum number of retries and any other conditions under which a retry should occur. Here's an example from the documentation on how to use a retry policy with an op:

python from dagster import op, RetryPolicy  @op(retry_policy=RetryPolicy(max_retries=3)) def my_op(context):     # Your op logic here     pass

In this example,

my_op

will be retried up to three times if it fails. However, this does not inherently track an increment value within the op. If you need to track an increment value that persists across retries or runs, you would likely need to store this value externally, such as in a database or a file, and then read it when the op starts. This way, if the op fails and is retried, it can read the last known increment value and proceed from there. Remember that the specifics of how to implement this depend on your particular use case and the nature of the increment value you're tracking. The documentation does not provide a specific pattern for this scenario, so you would need to design a custom solution that fits your requirements.