due to SQLAlchemy < 2.0 for dagster, I've tried...
# ask-community
c
due to SQLAlchemy < 2.0 for dagster, I've tried to create use a separate control-plane (dagster) and data-plane (ibis, duckdb, bunch of stuff). I am writing a custom IOManager that needs stuff in the data-plane virtual environment. however, since this has to be imported by dagster to setup the Definition, I'm getting an import error is this a known issue or is there some workaround? (I am new to dagster)
image.png
so I have two venvs,
venv_dag
with just dagster/dagit installed, and
venv
with everything else
Copy code
The above exception was caused by the following exception:
ModuleNotFoundError: No module named 'ibis'
  File "/Users/cody/repos/.../venv_dag/lib/python3.11/site-packages/dagster/_core/code_pointer.py", line 135, in load_python_module
    return importlib.import_module(module_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cody/.asdf/installs/python/3.11.3/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/cody/repos/.../dagster-demo/dagster_project/__init__.py", line 4, in <module>
    from .resources import table_io_manager
  File "/Users/cody/repos/.../dagster-demo/dagster_project/resources/__init__.py", line 1, in <module>
    from .table_io_manager import TableIOManager
  File "/Users/cody/repos/.../dagster-demo/dagster_project/resources/table_io_manager.py", line 1, in <module>
    import ibis
I don't really see a way for them to not coexist because I need
from dagster import ConfigurableIOManager
and
import ibis
in the same Python file
z
yeah I don't think you can natively execute ops in an environment which doesn't have dagster installed, because the op itself (as well as IO manager) is a dagster construct. if you need to execute fully separated code like this you'll need to go down the route of launching an external job on k8s, ECS, or docker, and having your op monitor the job for completion
👍 1