https://dagster.io/ logo
Title
j

James Rosenthal

02/08/2023, 6:40 PM
Hi team! I am trying to configure an Airbyte connection between a Postgres Source and a Snowflake Destination using Dagster. I have created a file in from the template https://github.com/airbytehq/open-data-stack/tree/main/dagster (and instructions here: https://airbyte.com/tutorials/configure-airbyte-with-python-dagster) updated to use my own source and destination in the file
load_snowflake.py
. When I run:
dagster-airbyte check --module assets_modern_data_stack.assets.load_snowflake:airbyte_reconciler
I keep on getting the error:
ModuleNotFoundError: No module named 'assets_modern_data_stack'
Does anyone know why I might be encountering this error and how I can fix it? I made sure that I was running the command from the same directory that has
assets_modern_data_stack
. Thanks!
Full error message:
Loading module...
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.10/site-packages/dagster_managed_elements/cli.py", line 43, in get_reconcilable_objects_from_module
module = importlib.import_module(module_str)
File "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'assets_modern_data_stack'
It seems to get further when I instead run:
python3 -m assets_modern_data_stack.utils.setup_airbyte
But now I am getting an error that my connection to the source Postgres server is failing. These are the same parameters I used to connect successfully in the Airbyte UI, so I am not sure why they are not working here... checking the AWS settings to debug
b

ben

02/09/2023, 11:22 PM
Hi James, one thing that might help with the
dagster-airbyte check
error you’re seeing is adding the
--working-directory .
argument, this will append the current directory to the Python path so it can find modules located where you’re running the command
👍 1
e.g.
dagster-airbyte check --module foobar --working-directory .