I'm running a daily job to update information in an external system using data from a

SourceAsset

. I'd like to track which rows of information have been sent to the external system or have been updated since the last time the job was run. I'd like to use dagster to track this state information. I feel like I need to use an

asset

for this tracking, but this asset would need to be used as an input and output of my job (since the job will alter this state). Is this possible or is there a better way of keeping this state information with dagster? To do this without dagster, I'd manually read and write from a state file in my job but I'd prefer to let dagster handle this IO.

The first way I would think to do this would be to include metadata on your materialization for your asset that says what it wrote to the external system, then load it next time using

context.instance.get_latest_materialization_event(asset_key)

thanks, I could see how this could work! Can you see a way where this state data is stored in my own resource, like a snowflake table?

I think that could work as well

Actually, I don't think I can pull it back in as a

SourceAsset

because both have the same

AssetKey

. Is there a way to select a table as a

SourceAsset

but use a different key? I'm not seeing this in the docs

Is there a way to select a table as a

SourceAsset

but use a different key?

I'm not 100% following what you mean by this. Do you have a short code example of how you'd like this to work?

yeah so I'm creating an

Asset

that I'd like to bring back in as a

SourceAsset

(to be able to store and update a job's state) but it's not working because they both have the same

AssetKey

, so I'd need something like this:

my_current_state = SourceAsset(
    key="my_current_state", # a unique AssetKey for dagster
    table_name="state", # the table name in Snowflake, which is a dagster asset
)

Ah I see - alas, I can't think of a great way to get around this. One way could me to include metadata on your asset and write a custom IO manager that chooses the path based onthe metadata?