https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
d

Dane Linssen

05/03/2023, 7:23 AM
Hey crew!! I wanted to report back on our experience when migrating from
1.1.21
to
1.2.0
: • According to the migration document, migrating to
1.2.0
adds primary key id columns to the
kvs
,
daemon_heartbeats
and
instance_info
tables: https://github.com/dagster-io/dagster/blob/master/MIGRATION.md#database-migration • We followed this manual: https://docs.dagster.io/deployment/guides/kubernetes/how-to-migrate-your-instance • After a successful job in k8s …
Copy code
kubectl get po -n dagster
NAME                                                              READY   STATUS                       RESTARTS   AGE
...
dagster-dagit-instance-migrate-lmgz9                              0/1     Completed                    0          2m24s
...
… we still don’t have primary key id columns in the
kvs
,
daemon_heartbeats
and
instance_info
tables. These are the logs:
Copy code
Updating run storage...
Skipping already applied data migration: run_partitions
Skipping already applied data migration: run_repo_label_tags
Skipping already applied data migration: bulk_action_types
Updating event storage...
Skipping already applied data migration: asset_key_index_columns
Updating schedule storage...
Skipping already applied migration: schedule_jobs_selector_id
It skips steps as “they have already been applied”. This is not currently an issue for us, just wanted to report back as this might be valuable information for you 🙂. Thanks for everything that you’re doing! Cheers!
p

prha

05/03/2023, 8:21 PM
Hi Dane. That’s very odd! Can you share the alembic hash in the
alembic_version
table? And can you confirm which storage you’re using?
(The skip logs refer to data migrations…. I would assume that the schema migrations would still be applied if necessary)
t

Tim van Druenen

05/04/2023, 9:08 AM
Hi @prha, I'm taking over from @Dane Linssen to avoid him being a proxy. To answer your questions:
Copy code
SELECT * from public.alembic_version;
 version_num
--------------
 d9092588866f
(1 row)
About which storage we're using, I guess you want to know which database (version) we're running. I've tried the db migration with two different deployments of dagster with two different databases: 1. very basic Postgres 11 (deployed by Zalando Operator) 2. Postgres 14.6 incl. pgBouncer (deployed by Stackgres Operator) All in Kubernetes with active/standby (replication) enabled. I used this query to verify if there are primary keys available:
Copy code
select tab.table_schema,
       tab.table_name
from information_schema.tables tab
left join information_schema.table_constraints tco 
          on tab.table_schema = tco.table_schema
          and tab.table_name = tco.table_name 
          and tco.constraint_type = 'PRIMARY KEY'
where tab.table_type = 'BASE TABLE'
      and tab.table_schema not in ('pg_catalog', 'information_schema')
      and tco.constraint_name is null
order by table_schema,
         table_name;
The output before and after the migration was the same:
Copy code
table_schema |    table_name
--------------+-------------------
 public       | daemon_heartbeats
 public       | instance_info
 public       | kvs
(3 rows)
You can see the full output of the migration pod attached. Thanks for your help!
p

prha

05/04/2023, 1:23 PM
Hi Tim. Thanks for the report! For postgres, I’ve confirmed that the id columns are created but a primary key constraint was not created. I can get a patch in for next week’s release. In the meantime, are you seeing any issues in the operation of dagit? The main motivation behind making sure all tables had primary keys was because of some errors operating dagster on a MySQL server that I don’t think apply here…
t

Tim van Druenen

05/05/2023, 7:31 AM
Hi! Thanks for picking this up so quick. No, we didn't had any issues. I noticed it while performing a manual check in the Postgres database. I verified if there were any tables without primary keys, because I wanted to enable Logical Replication to be able to perform a (near) zero downtime major upgrade of the database. I noticed the three tables, but I also remembered a comment in the release notes of dagster saying they should have primary keys after the latest database migration, that's why we came here to ask 🙂
p

prha

05/05/2023, 1:02 PM
Got it. Really appreciate you raising it! We’ll get this in for
1.3.4
daggy 3d 2