https://dagster.io/ logo
Powered by
Title
y

Yoav Gilor

05/22/2023, 12:44 PM
Hey there! Can anyone help with this error I get when executing a pipeline? 
sqlalchemy.exc.ProgrammingError: (mysql.connector.errors.ProgrammingError) Failed processing pyformat-parameters; Python 'pendulum' cannot be converted to a MySQL type
[SQL: UPDATE runs SET `status`=%(status)s, run_body=%(run_body)s, update_timestamp=%(update_timestamp)s, start_time=%(start_time)s WHERE runs.run_id = %(run_id_1)s]
[parameters: {'status': 'STARTED', 'run_body': '{"__class__": "PipelineRun", "asset_selection": {"__frozenset__": []}, "execution_plan_snapshot_id": "00fa4f5ffdc965990dadefa6471507a4a8a2396b", "ext ... (3958 characters truncated) ... rucor_FSOS_By_SceneType_By_SKU_1608", "write_vanilla_calculations", "combine_kpi_results", "generate_result_hierarchy", "write_results"], "tags": {}}', 'update_timestamp': <Pendulum [2023-05-22T12:23:58.215929+00:00]>, 'start_time': 1684758238.215929, 'run_id_1': '12ef4820-d9eb-4975-9b17-ba09b8ea247b'}]
It's happening in the execute_in_process and execute_job methods when it's trying to update the runs table in the database. I see here : 
dagster/_core/storage/runs/sql_run_storage.py:206
that the update_timestamp field is set as a Pendulum object. but from some reason the MySQL cannot handle that. I recently installed pendulum==1.4.4 in my environment to solve a different issue, perhaps it's related? Thanks!
o

owen

05/22/2023, 8:42 PM
Hi @Yoav Gilor! My suspicion is that this has something to do with your mysql-connector version, as this is what is in charge of doing this conversion. Pinning the pendulum version was likely the catalyst for whatever went wrong, but dagster is compatible with (and tested against) pendulum versions < 2 and > 2, so in theory that should still work.
y

Yoav Gilor

05/23/2023, 9:20 AM
Hey @owen, Thanks for the reply! Which module are you referring to? mysql-connector-python or pymysql? I see I have both of them
o

owen

05/23/2023, 3:51 PM
mysql-connector-python
y

Yoav Gilor

05/28/2023, 10:36 AM
Hey @owen, Can you recommend a version which runs well with Dagster 1.3.2 ? I've tried some and either got the same error or got other version mismatch errors
o

owen

05/30/2023, 4:39 PM
I have 
mysql-connector-python==8.0.31
-- if you do a fresh install of dagster in an empty virtualenv, do you still get this issue?
#dagster-supportJoin Slack
Powered by