https://dagster.io/ logo
#ask-community
Title
# ask-community
p

Pablo Beltran

01/30/2023, 10:01 PM
Hey there we are seeing the following error on the tickers: sqlalchemy.exc.ArgumentError Running the jobs manually seems to work. Any suggestions on how to fix. We have tried restarting the daemon and dagit so that it picks up the latest image with no success.
d

daniel

01/30/2023, 10:04 PM
Hi Pablo - if you add a "sqlalchemy<2" pin or upgrade to the latest version of dagster i suspect this will go away - a breaking change came in the latest version of the sqlalchemy package
w

Wonjae Lee

01/30/2023, 10:17 PM
Hello Daniel, Thanks for the response. We indeed update all our images / helm chart version to 1.1.14 (which is the latest), restart all daemon / pods but still no luck on the scheduled jobs 😞 sqlalchemy.exc.ArgumentError: Column expression, FROM clause, or other columns clause element expected, got [Column(‘id’, Integer(), table=<runs>, primary_key=True, nullable=False), Column(‘run_body’, Text(), table=<runs>), Column(‘status’, String(length=63), table=<runs>), Column(‘create_timestamp’, DateTime(), table=<runs>,
d

daniel

01/30/2023, 10:17 PM
I see - can you share where exactly you're seeing that log message?
w

Wonjae Lee

01/30/2023, 10:17 PM
sure
Screen Shot 2023-01-30 at 2.17.39 PM.png
we deploy and serve Dagster through k8s argo
d

daniel

01/30/2023, 10:18 PM
Got it - so that error is happening in your user code deployment image. Are you certain that that image is on dagster 1.1.14 or has sqlalchemy pinned to < 2.0.0.?
w

Wonjae Lee

01/30/2023, 10:20 PM
yep so we explicitly use 1.1.14 on our 1. Helm chart version (checked that each pod has this ver) 2. Pull image dagster-k8s version with 1.1.14 (checked it downloads this ver) 3. Our custom image downloads (pip install) 1.1.14 for all dagster
d

daniel

01/30/2023, 10:20 PM
Any chance you could share the Dockerfile for your user code deployment?
If you have logs from building it, that might explain why the pin is being ignored
w

Wonjae Lee

01/30/2023, 10:24 PM
yeap here is the docker code image. So before updating below docker image, we couldn’t run the manual launch on Dagster website. After deploying this docker update, we were able to launch the job, but it fails at scheduled run
d

daniel

01/30/2023, 10:25 PM
Is there any chance that there's a different version of dagster referenced in that last line that installed
dagster_module/.
? Does that have a setup.py?
If that setup.py referenced a different version of dagster, it would overwrite the one earlier in the Dockerfile and match the symptoms that you're seeing
w

Wonjae Lee

01/30/2023, 10:26 PM
thanks for the suggestion. let me quickly check on this
in the docker image: pip3 freeze shows we are using < 2.0 on sqlalchemy SQLAlchemy==1.4.46
d

daniel

01/30/2023, 10:32 PM
I'm not sure what to make of that - that's an error that only pops up for people on SqlAlchemy 2, every other report I've seen was fixed when they downgraded back to 1.4.46
could there be multiple python environments in your image somehow?
w

Wonjae Lee

01/30/2023, 10:43 PM
let me double check on our entire flow.
Thanks Daniel for your help here. We were able to resolve the issue by restarting the specific pod includes our scheduled job codes (we restarted dagit and daemon pods but they were fine).
🎉 1