due to SQLAlchemy lt 2 0 for dagster I ve tried to create us

Question

due to SQLAlchemy lt 2 0 for dagster I ve tried to create use a separate control plane dagster and data plane ibis duckdb bunch of stuff I am writing a custom IOManager that needs stuff in the data plane virtual environment however since this has to be imported by dagster to setup the Definition I m getting an import error is this a known issue or is there some workaround I am new to dagster

Cody Peterson · Answer

image png

Cody Peterson · Answer

so I have two venvs `venv dag` with just dagster dagit installed and `venv` with everything else

Cody Peterson · Answer

```The above exception was caused by the following exception ModuleNotFoundError No module named ibis File Users cody repos venv dag lib python3 11 site packages dagster core code pointer py line 135 in load python module return importlib import module module name ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File Users cody asdf installs python 3 11 3 lib python3 11 importlib init py line 126 in import module return bootstrap gcd import name level package level ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File lt frozen importlib bootstrap gt line 1206 in gcd import File lt frozen importlib bootstrap gt line 1178 in find and load File lt frozen importlib bootstrap gt line 1149 in find and load unlocked File lt frozen importlib bootstrap gt line 690 in load unlocked File lt frozen importlib bootstrap external gt line 940 in exec module File lt frozen importlib bootstrap gt line 241 in call with frames removed File Users cody repos dagster demo dagster project init py line 4 in lt module gt from resources import table io manager File Users cody repos dagster demo dagster project resources init py line 1 in lt module gt from table io manager import TableIOManager File Users cody repos dagster demo dagster project resources table io manager py line 1 in lt module gt import ibis```

Cody Peterson · Answer

I don t really see a way for them to not coexist because I need `from dagster import ConfigurableIOManager` and `import ibis` in the same Python file

Zach · Answer

yeah I don t think you can natively execute ops in an environment which doesn t have dagster installed because the op itself as well as IO manager is a dagster construct if you need to execute fully separated code like this you ll need to go down the route of launching an external job on k8s ECS or docker and having your op monitor the job for completion