https://dagster.io/ logo
Join Slack
Powered by
Hello, I'm trying to test the dagster-shell integr...
# ask-community
n
Hello, I'm trying to test the dagster-shell integration and use the dagster_shell module to import functions, but I keep getting an error whenever I try to boot up dagster. This error says it cannot import functions from dagster_shell because it is partially initialized, most likely due to a circular import. So I tried to pip install dagster-shell manually rather than putting it as a dependency in setup.py in my project, but the most recent version of dagster-shell is 1.0.5, only compatible with dagster 1.0.5, but I have several other dependencies that I have working for dagster 1.3.5 at the moment. I apologize if this is just something silly, I'm still very new to using dagster.
d
Hey Noah - there's a yanked 1.0.5 dagster-shell in pypi, but the most recent version of dagster-shell is actually 0.19.9 (the version of dagster-shell that's compatible with dagster 1.3.5 is 0.19.5 - dagster is 1.0 but some of the integration libraries are pre-1.0)
n
Thank you so much, I'll try to use 0.19.5!
Alright, so it looks like I don't have any incompatibility problems between dagster and dagster-shell anymore. However, I am still getting the same error when I start up dagster, saying the module is partially initialized, most likely due to a circular import.
d
Can you share the stack trace of the error?
n
Copy code
UserWarning: Error loading repository location dagster_shell:dagster._core.errors.DagsterImportError: Encountered ImportError: `cannot import name 'create_shell_command_op' from partially initialized module 'dagster_shell' (most likely due to a circular import) (/Users/noahnewton/dev/dagster-old/dagster-shell/dagster_shell/__init__.py)` while importing module dagster_shell. Local modules were resolved using the working directory `/Users/noahnewton/dev/dagster-old/dagster-shell`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for workspace targets. 

Stack Trace:
  File "/Users/noahnewton/mambaforge/envs/dag3.10/lib/python3.10/site-packages/dagster/_grpc/server.py", line 270, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
  File "/Users/noahnewton/mambaforge/envs/dag3.10/lib/python3.10/site-packages/dagster/_grpc/server.py", line 119, in __init__
    loadable_targets = get_loadable_targets(
  File "/Users/noahnewton/mambaforge/envs/dag3.10/lib/python3.10/site-packages/dagster/_grpc/utils.py", line 47, in get_loadable_targets
    else loadable_targets_from_python_module(module_name, working_directory)
  File "/Users/noahnewton/mambaforge/envs/dag3.10/lib/python3.10/site-packages/dagster/_core/workspace/autodiscovery.py", line 35, in loadable_targets_from_python_module
    module = load_python_module(
  File "/Users/noahnewton/mambaforge/envs/dag3.10/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 140, in load_python_module
    raise DagsterImportError(

The above exception was caused by the following exception:
ImportError: cannot import name 'create_shell_command_op' from partially initialized module 'dagster_shell' (most likely due to a circular import) (/Users/noahnewton/dev/dagster-old/dagster-shell/dagster_shell/__init__.py)

Stack Trace:
  File "/Users/noahnewton/mambaforge/envs/dag3.10/lib/python3.10/site-packages/dagster/_core/code_pointer.py", line 135, in load_python_module
    return importlib.import_module(module_name)
  File "/Users/noahnewton/mambaforge/envs/dag3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/noahnewton/dev/dagster-old/dagster-shell/dagster_shell/__init__.py", line 3, in <module>
    from . import assets
  File "/Users/noahnewton/dev/dagster-old/dagster-shell/dagster_shell/assets.py", line 2, in <module>
    from dagster_shell import create_shell_command_op, create_shell_script_op, execute_shell_command, execute_shell_script

  warnings.warn(
d
huh, and is it possible to share the code that's hitting that?
n
Sure, it's from the dagster docs:
Copy code
from dagster import graph, file_relative_path, OpExecutionContext, op
from dagster_shell import create_shell_command_op, create_shell_script_op, execute_shell_command, execute_shell_script

@graph
def my_graph():
    a = create_shell_command_op('echo "hello world"', name="a")
    a()

@graph
def my_graph2():
    a = create_shell_script_op(file_relative_path(__file__, "hello_world.sh"), name="a")
    a()

@op
def my_shell_op(context: OpExecutionContext, data: str):
    temp_file = "/tmp/data.txt"
    with open(temp_file, "w", encoding="utf-8") as temp_file_writer:
        temp_file_writer.write(data)
        execute_shell_command(f"cat {temp_file}", output_logging="STREAM", log=context.log)

@op
def my_shell_op2(context: OpExecutionContext, data: str):
    temp_file = "/tmp/echo_data.sh"
    with open(temp_file, "w", encoding="utf-8") as temp_file_writer:
        temp_file_writer.write(f"echo {data}")
        execute_shell_script(temp_file, output_logging="STREAM", log=context.log)
d
I don't see an 'assets.py' in dagster-shell, but it's referenced in your stack trace: https://github.com/dagster-io/dagster/tree/1.3.9/python_modules/libraries/dagster-shell/dagster_shell - what version do you have installed? I checked a couple of different releases and didn't see it there
i.e. I can't find this line in dagster anywhere:
Copy code
File "/Users/noahnewton/dev/dagster-old/dagster-shell/dagster_shell/__init__.py", line 3, in <module>
    from . import assets
Are you sure that's not something that you somehow created in your code?
n
I'm currently on dagster 1.3.5. That line is in one of the files that was automatically created when I first created the project. Here is the full file:
Copy code
from dagster import Definitions, load_assets_from_modules

from . import assets

all_assets = load_assets_from_modules([assets])

defs = Definitions(
    assets=all_assets,
)
d
It's looking like that might have somehow gotten created within the dagster_shell folder?
n
Yes
d
I don't think that folder is intended to be modified
Your definitions would go in a separate package
n
When I created the project, named dagster-shell, it created that folder inside, dagster_shell, with the assets and init python files. On the previous dagster tutorials I did, they had me change some things inside of the corresponding assets.py files that were created. If the init file shouldn't be there, where should it go?
d
I would advise against creating a project that has the same name as a dagster integration library, i think that might be one of the things that is confusing it
n
Oh ok, I should try starting over with a different project name?
d
Yeah, try that
n
Alright
It appears to be loading into dagster now without any errors. Thanks for the help!
condagster 1
2 Views
Open in Slack
PreviousNext
Powered by