Hi all, I’m running dagster on k8s (aws EKS) and trying to upgrade dagster version form 0.10.4 to 0.12.8. my setup includes aws RDS postgres (for dagster backend). it was deployed with helm chart, where dagit and the daemon separated from user-code. after deployed v0.12.8 using same RDS postgres, I’ve ran dagster instance migration job, which ends successfully. I see all jobs history and I’ve tested pipelines/sensor/schedules - those seems to work just fine. The only problem is that I’m missing

AssetMaterialization

history, prior to the upgrade. after the upgrade, when running related pipeline, dagit ui does shows AssetMaterialization visualisation and it history (but not prior to upgrade). I’m not sure where exactly the issue, is it just visualisation issue or those records are missing in the postgres db. I will appreciate any help.

cc @prha

Hey Roman… The history of your asset materializations is still preserved, but we added an asset index table in the DB to optimize the catalog queries. We introduced a lazy migration, but maybe not everything got transferred over? You can try running

dagster instance reindex

and see if that restores the asset entries. Caution, this might take a long time to run. The migration notes are here, from the

0.11.0

release: https://github.com/dagster-io/dagster/blob/master/MIGRATION.md#action-required-asset-storage

thank you for a prompt response @prha. I missed some context earlier that might be relevant. We decided to reshape helm chart, then the one we had for v0.10.4. We created a separate helm charts for dragster digit/daemon and additional chart for user-code. as mentioned these are separate charts, then we had for v0.10.4 chart. I dont think it got to do with the missing AssetMaterialization history, but thought it worth to mention to get the whole picture. hypothesis: could be that Postgres db has some kind dependency with helm charts when using different charts it break the history? any way it my speculation. re your suggestion - I gave it a try, on my staging environment. I had one asset record prior to upgrade. I’ve moved to v0.12.8 and ran

instance migrate

command and right after that

instance reindex

(in my case took 1s) - here is the output for the latest:

# dagster instance reindex
$DAGSTER_HOME: /opt/dagster/dagster_home

Checking for reindexing...
Skipping already reindexed summary: asset_key_table
Skipping already reindexed summary: asset_key_index_columns
Done.

I got the same results (see attached screensnips) - I dont see asset history prior to upgrade.  From migration notes for v0.11.0 says

you can either run dagster instance reindex or execute the appropriate pipelines to materialize those assets again.

- just to clarify - the second part mentions to run appropriate pipeline - as far as I understand it’s relevant to new run and it won’t solve missing history - is that correct?

Hmm, I don’t think the helm chart would make a difference here in a way that would affect assets and not runs. Was your run history preserved (e.g. could you view the runs created prior to upgrading)?

yes. I can see all run history back to Jun. just missing asset history 🤷.

we should schedule a debugging session…. reached out to you in DMs