Hey crew!! I wanted to report back on our experience when migrating from

1.1.21

to

1.2.0

: • According to the migration document, migrating to

1.2.0

adds primary key id columns to the

kvs

,

daemon_heartbeats

and

instance_info

tables: https://github.com/dagster-io/dagster/blob/master/MIGRATION.md#database-migration • We followed this manual: https://docs.dagster.io/deployment/guides/kubernetes/how-to-migrate-your-instance • After a successful job in k8s …

kubectl get po -n dagster
NAME                                                              READY   STATUS                       RESTARTS   AGE
...
dagster-dagit-instance-migrate-lmgz9                              0/1     Completed                    0          2m24s
...

… we still don’t have primary key id columns in the

kvs

,

daemon_heartbeats

and

instance_info

tables. These are the logs:

Updating run storage...
Skipping already applied data migration: run_partitions
Skipping already applied data migration: run_repo_label_tags
Skipping already applied data migration: bulk_action_types
Updating event storage...
Skipping already applied data migration: asset_key_index_columns
Updating schedule storage...
Skipping already applied migration: schedule_jobs_selector_id

It skips steps as “they have already been applied”. This is not currently an issue for us, just wanted to report back as this might be valuable information for you 🙂. Thanks for everything that you’re doing! Cheers!

Hi Dane. That’s very odd! Can you share the alembic hash in the

alembic_version

table? And can you confirm which storage you’re using?

Hi @prha, I'm taking over from @Dane Linssen to avoid him being a proxy. To answer your questions:

SELECT * from public.alembic_version;
 version_num
--------------
 d9092588866f
(1 row)

About which storage we're using, I guess you want to know which database (version) we're running. I've tried the db migration with two different deployments of dagster with two different databases: 1. very basic Postgres 11 (deployed by Zalando Operator) 2. Postgres 14.6 incl. pgBouncer (deployed by Stackgres Operator) All in Kubernetes with active/standby (replication) enabled. I used this query to verify if there are primary keys available:

select tab.table_schema,
       tab.table_name
from information_schema.tables tab
left join information_schema.table_constraints tco 
          on tab.table_schema = tco.table_schema
          and tab.table_name = tco.table_name 
          and tco.constraint_type = 'PRIMARY KEY'
where tab.table_type = 'BASE TABLE'
      and tab.table_schema not in ('pg_catalog', 'information_schema')
      and tco.constraint_name is null
order by table_schema,
         table_name;

The output before and after the migration was the same:

table_schema |    table_name
--------------+-------------------
 public       | daemon_heartbeats
 public       | instance_info
 public       | kvs
(3 rows)

You can see the full output of the migration pod attached. Thanks for your help!

Hi Tim. Thanks for the report! For postgres, I’ve confirmed that the id columns are created but a primary key constraint was not created. I can get a patch in for next week’s release. In the meantime, are you seeing any issues in the operation of dagit? The main motivation behind making sure all tables had primary keys was because of some errors operating dagster on a MySQL server that I don’t think apply here…

Hi! Thanks for picking this up so quick. No, we didn't had any issues. I noticed it while performing a manual check in the Postgres database. I verified if there were any tables without primary keys, because I wanted to enable Logical Replication to be able to perform a (near) zero downtime major upgrade of the database. I noticed the three tables, but I also remembered a comment in the release notes of dagster saying they should have primary keys after the latest database migration, that's why we came here to ask 🙂

Got it. Really appreciate you raising it! We’ll get this in for

1.3.4