https://dagster.io/ logo
Powered by
c

Cam Peterson

01/13/2020, 11:21 PM
I’m just following the Cereal tutorial
m

max

01/13/2020, 11:21 PM
can i see more context on where this error is occuring?
the core devs don't use 
conda
very much -- it's possible that one of my recent changes doesn't play nicely with it
(if you can just copy and paste more of the output)
c

Cam Peterson

01/13/2020, 11:22 PM
Copy code
(dagster)  ✘  ~/projects/level2/mipu/mipu-dagster  dagit -f ./hello_cereal.py -n hello_cereal_pipeline                                                                                                                                                                               <<<
/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagster/core/serdes/__init__.py:199: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read <https://msg.pyyaml.org/load> for full details.
  config_dict = yaml.load(self.config_yaml)
Traceback (most recent call last):
  File "/Users/campeterson/anaconda3/envs/dagster/bin/dagit-cli", line 8, in <module>
    sys.exit(main())
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagit/cli.py", line 110, in main
    cli = create_dagit_cli()
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagit/cli.py", line 21, in create_dagit_cli
    return ui(auto_envvar_prefix='DAGIT')  # pylint: disable=no-value-for-parameter
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagit/cli.py", line 72, in ui
    host_dagit_ui(handle, host, port, storage_fallback, reload_trigger)
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagit/cli.py", line 78, in host_dagit_ui
    instance = DagsterInstance.get(storage_fallback)
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 151, in get
    return DagsterInstance.from_config(fallback_storage)
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 167, in from_config
    return DagsterInstance.from_ref(instance_ref)
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 176, in from_ref
    run_storage=instance_ref.run_storage,
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagster/core/instance/ref.py", line 143, in run_storage
    return self.run_storage_data.rehydrate()
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagster/core/serdes/__init__.py", line 203, in rehydrate
    return klass.from_config_value(self, result.value, **constructor_kwargs)
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagster/core/storage/runs/sqlite/sqlite_run_storage.py", line 34, in from_config_value
    return SqliteRunStorage.from_local(inst_data=inst_data, **dict(config_value, **kwargs))
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagster/core/storage/runs/sqlite/sqlite_run_storage.py", line 48, in from_local
    stamp_alembic_rev(alembic_config, conn)
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagster/core/storage/sql.py", line 35, in stamp_alembic_rev
    stamp(alembic_config, rev)
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/alembic/command.py", line 545, in stamp
    script = ScriptDirectory.from_config(config)
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/alembic/script/base.py", line 149, in from_config
    hook_config=config.get_section("post_write_hooks", {}),
  File "/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/alembic/script/base.py", line 70, in __init__
    "scripts folder." % dir
alembic.util.exc.CommandError: Path doesn't exist: '/Users/campeterson/anaconda3/envs/dagster/lib/python3.7/site-packages/dagster/core/storage/runs/sqlite/alembic/'.  Please use the 'init' command to create a new scripts folder.
I started out not using Conda, but had the same issue.
Copy code
dagster pipeline execute -f hello_cereal.py -n hello_cereal_pipeline
^ works fine
Copy code
dagit -f ./hello_cereal.py -n hello_cereal_pipeline
^ is the problem
I wondered if my Python env was messed up, so I decided to try it inside a Conda env.
It was strange to see the YAML error as I’d used the other flags
m

max

01/13/2020, 11:26 PM
the YAML warning is unrelated i believe
ok, thank you -- i am going to try to reproduce this, and i suspect we have made an error in our packaging
c

Cam Peterson

01/13/2020, 11:29 PM
Okay, awesome. Thanks Max
m

max

01/13/2020, 11:34 PM
thanks for this, this is indeed a packaging mistake
you can work around by cloning the repo from github https://github.com/dagster-io/dagster and then running 
pip install -e
to install dagit -- we will get a patch release out to fix this
thank you again for the report
c

Cam Peterson

01/13/2020, 11:36 PM
Yes, for sure. Glad to help
@max sorry, one more after installing from Github (outside Conda)…
(rechecking to make sure I’m reporting the right thing)
Copy code
dagit -p 3333
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (PyYAML 5.3 (/usr/local/lib/python3.7/site-packages), Requirement.parse('PyYAML<5,>=3.10'), {'dagster'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/dagit", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3251, in <module>
    @_call_aside
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3235, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3264, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 585, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 598, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 791, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (PyYAML 5.3 (/usr/local/lib/python3.7/site-packages), Requirement.parse('PyYAML<5,>=3.10'), {'dagster'})
So I dropped my Pyyaml version back to 3.13
then I got this:
Copy code
dagit -p 3333
Traceback (most recent call last):
  File "/usr/local/bin/dagit", line 11, in <module>
    load_entry_point('dagit', 'console_scripts', 'dagit')()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/campeterson/code/com.github/dagster-io/dagster/python_modules/dagit/dagit/dagit.py", line 12, in <module>
    from .cli import ui
  File "/Users/campeterson/code/com.github/dagster-io/dagster/python_modules/dagit/dagit/cli.py", line 15, in <module>
    from .app import create_app
  File "/Users/campeterson/code/com.github/dagster-io/dagster/python_modules/dagit/dagit/app.py", line 10, in <module>
    from dagster_graphql.implementation.pipeline_execution_manager import (
ImportError: cannot import name 'QueueingSubprocessExecutionManager' from 'dagster_graphql.implementation.pipeline_execution_manager' (/usr/local/lib/python3.7/site-packages/dagster_graphql/implementation/pipeline_execution_manager.py)
This is all after installing 
dagit
from source (but 
dagster
from 
pip
When I install both from source, I just end up with the 
QueueingSubprocessExecutionManager
error mentioned above (no pyyaml version issue)
m

max

01/14/2020, 12:25 AM
gotcha -- you will probably also need to install dagster_graphql from source
i am about to release 0.6.6.post0 0.6.7.post0, which should resolve the issue
if you don't mind, i'd love that you confirm that installing from pypi this problem is resolved for you
c

Cam Peterson

01/14/2020, 6:35 AM
0.6.7post0 is working. This was from a fresh install on a different computer. Dagster and Dagit installed from pip
👍 1
Powered by