https://dagster.io/ logo
Powered by
Title
m

mrdavidlaing

02/22/2022, 8:06 PM
After upgrading to 0.14.1 we've started to see errors like the below frequently when browsing Dagit (especially the new 
instance/overview
page) Is this a know issue; or something to do with our specific set of dependencies? 
AttributeError: 'Select' object has no attribute 'subquery'
Traceback (most recent call last):
  ...snip...
  File "/Users/dlaing/workspace/tanzu-dm/.venv/lib/python3.7/site-packages/dagster/core/storage/runs/sql_run_storage.py", line 312, in _runs_query
    subquery = base_query.subquery()
graphql.error.located_error.GraphQLLocatedError: 'Select' object has no attribute 'subquery'
p

prha

02/22/2022, 8:07 PM
I haven’t seen this before… What run storage implementation are you using?
m

mrdavidlaing

02/22/2022, 8:09 PM
a temporary SQLite one: 
run_storage:
  module: dagster.core.storage.runs
  class: SqliteRunStorage
  config:
    base_dir: /Users/dlaing/workspace/tanzu-dm/tmp/dagster_home/history/
I just deleted 
/Users/dlaing/workspace/tanzu-dm/tmp/dagster_home
and then restarted (to ensure no left over state version issues) - still getting the same error
p

prha

02/22/2022, 8:52 PM
can you also share what version of sqlite you have installed?
a

alex

02/22/2022, 8:54 PM
and 
SQLAlchemy
m

mrdavidlaing

02/22/2022, 8:57 PM
$ pip freeze | egrep 'SQL' 
SQLAlchemy==1.4.31
$ sqlite3 -version
3.36.0 2021-06-18 18:58:49 d24547a13b6b119c43ca2ede05fecaa707068f18c7430d47fc95fb5a2232aapl
p

prha

02/23/2022, 12:23 AM
Currently stumped by this one… I can’t repro, even when pinning to the same sqlite version / sqlalchemy version
m

mrdavidlaing

02/23/2022, 7:38 AM
Would jumping on a screenshare help? (We can repo on 2 separate Mac's, so I'm thinking it might be a "set of conflicting dependancy" problem). Unfortunately its an internal codebase so we can't share publicly.
a

alex

02/23/2022, 4:30 PM
i think 
sqlite3
(in shell) might not resolve to the same as what 
python
is using, so could double check in 
python
quick: 
>>> import sqlite3
>>> sqlite3.version
'2.6.0'
>>> sqlite3.sqlite_version
'3.32.3'
p

prha

02/23/2022, 4:33 PM
The other thing I can think of is to pass over the version of python you’re using (I tried repro-ing with 
3.7.12
) and maybe a 
requirements.txt
file (minus the private packages)
m

mrdavidlaing

02/26/2022, 9:36 AM
$ pipenv run python                                                                                                                                                                                         
Python 3.7.9 (default, Sep 22 2021, 10:42:20) 
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.version
'2.6.0'
>>> sqlite3.sqlite_version
'3.37.2'
>>>
Aha! I've just realised that due to some pip shenanigans; we actually have 
SQLAlchemy==1.3.15
installed
If I upgrade to 
SQLAlchemy==1.4.0
the problem goes away...
It looks like the dependency chain for 
SQLAlchemy
comes through 
dagster
-> 
alembic
(source); although 
alembic
only seems to specify 
SQLAlchemy>=1.3.0
(source) 
dagit==0.14.1
  - dagster [required: ==0.14.1, installed: 0.14.1]
    - alembic [required: >=1.2.1,<1.7.0,!=1.6.3, installed: 1.6.5]
      - SQLAlchemy [required: >=1.3.0, installed: 1.3.15]
Since SQLAlchemy < 1.4.0 appears to causes this error; I think one of the packages should have its requirements bumped to 
SQLAlchemy>=1.4.0
Just not sure if this should be 
dagster
or 
alembic
WDYT?
a

alex

02/28/2022, 5:37 PM
if we are using code in 
dagster
that is only available in > 1.4 we should probably improve our pin
not critical but did you find any changlog / documentation about 1.4 being when they added 
subquery
on 
Select
?
https://github.com/dagster-io/dagster/pull/6835?no-redirect=1
m

mrdavidlaing

03/01/2022, 10:11 PM
I think it was part of the new 1.4 API changes - Selectables, Tables, FROM objects — SQLAlchemy 1.4 Documentation
https://docs.sqlalchemy.org/en/14/core/selectable.html#sqlalchemy.sql.expression.SelectBase.subquery
p

prha

03/01/2022, 10:19 PM
I have a PR (https://github.com/dagster-io/dagster/pull/6841) that should gate this type of query based on what version of SQLAlchemy you have. A pin of SQLAlchemy conflicts with some version of airflow that we’re still supporting…
should be able to get this into the next release
m

mrdavidlaing

03/02/2022, 10:47 PM
Thanks for chasing this down - especially given the dependent hell it seems to bumped into!
#dagster-supportJoin Slack
Powered by