https://dagster.io/ logo
Title
a

Abhishek Sawant

11/08/2022, 2:34 PM
Hello, I'm trying to run redshift query from op >job >repository from the example. but i'm getting below error:
<path>/lib/python3.9/site-packages/dagster/_core/workspace/context.py:538: UserWarning: Error loading repository location dagster_redshift.repository:AssertionError
....
dagster_redshift/op/copy.py", line 76, in <module>
    assert example_redshift_op(context) == [(1,)]
The query is running but it is giving above error, below is the sample code (same from the example)
from dagster import build_op_context, op
from dagster_aws.redshift import redshift_resource

@op(required_resource_keys={'redshift'})
def example_redshift_op(context):
    return context.resources.redshift.execute_query('SELECT 1', fetch_results=True)

redshift_configured = redshift_resource.configured({
    'host': '<http://my-redshift-cluster.us-east-1.redshift.amazonaws.com|my-redshift-cluster.us-east-1.redshift.amazonaws.com>',
    'port': 5439,
    'user': '<usr_id>',
    'password': '<pwd>',
    'database': 'dev',
})
context = build_op_context(resources={'redshift': redshift_configured})
assert example_redshift_op(context) == [(1,)]
y

yuhan

11/08/2022, 8:22 PM
does
example_redshift_op(context)
give any result?
a

Abhishek Sawant

11/09/2022, 4:32 AM
yes it seems to be- below is the output (did it with select now())-
% dagit
Using temporary directory <path>/tmpch9g1ksk for storage. This will be removed when dagit exits.
To persist information across sessions, set the environment variable DAGSTER_HOME to a directory to use.

0it [00:00, ?it/s]
0it [00:00, ?it/s]
2022-11-09 09:59:02 +0530 - dagster - INFO - resource:redshift - Executing query 'select now();'
<path>/env_redshift/lib/python3.9/site-packages/dagster/_core/workspace/context.py:538: UserWarning: Error loading repository location dagster_redshift.repository:AssertionError

Stack Trace:
  File "<path>/env_redshift/lib/python3.9/site-packages/dagster/_grpc/server.py", line 230, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
  File "<path>/env_redshift/lib/python3.9/site-packages/dagster/_grpc/server.py", line 104, in __init__
    loadable_targets = get_loadable_targets(
  File "<path>/env_redshift/lib/python3.9/site-packages/dagster/_grpc/utils.py", line 53, in get_loadable_targets
    else loadable_targets_from_python_package(package_name, working_directory)
  File "<path>/env_redshift/lib/python3.9/site-packages/dagster/_core/workspace/autodiscovery.py", line 49, in loadable_targets_from_python_package
    module = load_python_module(
  File "<path>/env_redshift/lib/python3.9/site-packages/dagster/_core/code_pointer.py", line 136, in load_python_module
    return importlib.import_module(module_name)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<path>/dagster_redshift/repository.py", line 2, in <module>
    from <http://dagster_redshift.jobs|dagster_redshift.jobs> import copy_tb_job
  File "<path>/dagster_redshift/jobs.py", line 2, in <module>
    from dagster_redshift.op.copy import example_redshift_op
  File "<path>/dagster_redshift/op/copy.py", line 16, in <module>
    assert example_redshift_op(context) == [(1,)]

  warnings.warn(
d

Dagster Jarred

11/18/2022, 4:05 AM
are you still having trouble with this?
a

Abhishek Sawant

11/18/2022, 7:08 AM
No, I'm using create_engine function-
from sqlalchemy import create_engine
conn = create_engine(f"postgresql+psycopg2://{username}:{password}@{server}:{port}/{db}")
below is called in ops-
conn = get_redshift_creds().raw_connection()