Hey there we are seeing the following error on the...
# ask-community
p
Hey there we are seeing the following error on the tickers: sqlalchemy.exc.ArgumentError Running the jobs manually seems to work. Any suggestions on how to fix. We have tried restarting the daemon and dagit so that it picks up the latest image with no success.
d
Hi Pablo - if you add a "sqlalchemy<2" pin or upgrade to the latest version of dagster i suspect this will go away - a breaking change came in the latest version of the sqlalchemy package
w
Hello Daniel, Thanks for the response. We indeed update all our images / helm chart version to 1.1.14 (which is the latest), restart all daemon / pods but still no luck on the scheduled jobs 😞 sqlalchemy.exc.ArgumentError: Column expression, FROM clause, or other columns clause element expected, got [Column(‘id’, Integer(), table=<runs>, primary_key=True, nullable=False), Column(‘run_body’, Text(), table=<runs>), Column(‘status’, String(length=63), table=<runs>), Column(‘create_timestamp’, DateTime(), table=<runs>,
d
I see - can you share where exactly you're seeing that log message?
w
sure
Screen Shot 2023-01-30 at 2.17.39 PM.png
we deploy and serve Dagster through k8s argo
d
Got it - so that error is happening in your user code deployment image. Are you certain that that image is on dagster 1.1.14 or has sqlalchemy pinned to < 2.0.0.?
w
yep so we explicitly use 1.1.14 on our 1. Helm chart version (checked that each pod has this ver) 2. Pull image dagster-k8s version with 1.1.14 (checked it downloads this ver) 3. Our custom image downloads (pip install) 1.1.14 for all dagster
d
Any chance you could share the Dockerfile for your user code deployment?
If you have logs from building it, that might explain why the pin is being ignored
w
yeap here is the docker code image. So before updating below docker image, we couldn’t run the manual launch on Dagster website. After deploying this docker update, we were able to launch the job, but it fails at scheduled run
d
Is there any chance that there's a different version of dagster referenced in that last line that installed
dagster_module/.
? Does that have a setup.py?
If that setup.py referenced a different version of dagster, it would overwrite the one earlier in the Dockerfile and match the symptoms that you're seeing
w
thanks for the suggestion. let me quickly check on this
in the docker image: pip3 freeze shows we are using < 2.0 on sqlalchemy SQLAlchemy==1.4.46
d
I'm not sure what to make of that - that's an error that only pops up for people on SqlAlchemy 2, every other report I've seen was fixed when they downgraded back to 1.4.46
could there be multiple python environments in your image somehow?
w
let me double check on our entire flow.
Thanks Daniel for your help here. We were able to resolve the issue by restarting the specific pod includes our scheduled job codes (we restarted dagit and daemon pods but they were fine).
🎉 1