Hi guys, I'm having some trouble setting up a sche...
# announcements
a
Hi guys, I'm having some trouble setting up a schedule
Copy code
dagster schedule up --preview -y repository.yaml
Traceback (most recent call last):
  File "/usr/local/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/dagster/cli/__init__.py", line 38, in main
    cli(obj={})  # pylint:disable=E1123
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/dagster/cli/schedule.py", line 133, in schedule_up_command
    return execute_up_command(preview, kwargs, click.echo)
  File "/usr/local/lib/python3.6/dist-packages/dagster/cli/schedule.py", line 149, in execute_up_command
    print_changes(scheduler_handle, repository, instance, print_fn, preview=preview)
  File "/usr/local/lib/python3.6/dist-packages/dagster/cli/schedule.py", line 55, in print_changes
    changeset = scheduler_handle.get_change_set(repository, instance)
  File "/usr/local/lib/python3.6/dist-packages/dagster/core/scheduler/scheduler.py", line 125, in get_change_set
    schedules = instance.all_schedules(repository)
  File "/usr/local/lib/python3.6/dist-packages/dagster/core/instance/__init__.py", line 529, in all_schedules
    return self._schedule_storage.all_schedules(repository)
AttributeError: 'NoneType' object has no attribute 'all_schedules'
m
@sashank @prha
p
@Auster Cid do you have a schedule storage defined in your
dagster.yaml
?
a
yes, this is my local config
Copy code
scheduler:
    module: dagster_cron.cron_scheduler
    class: SystemCronScheduler
run_launcher:
    module: dagster_graphql.launcher
    class: RemoteDagitRunLauncher
    config:
        address: <http://localhost:3000>
dagit:
    execution_manager:
        max_concurrent_runs: 4
oh, my dagster_home env var wasnt set, sorry
but anyway, I get a different error when trying to run the same command on my EC2 dagit instance set up using dagster-aws
p
great! can you paste the error you’re getting?
a
yes, just a min
Copy code
/opt/dagster/app$ dagster schedule up --preview -y repository.yaml
Traceback (most recent call last):
  File "/opt/dagster/venv/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/cli/__init__.py", line 38, in main
    cli(obj={})  # pylint:disable=E1123
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/cli/schedule.py", line 133, in schedule_up_command
    return execute_up_command(preview, kwargs, click.echo)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/cli/schedule.py", line 143, in execute_up_command
    instance = DagsterInstance.get()
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/instance/__init__.py", line 186, in get
    return DagsterInstance.from_config(_dagster_home())
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/instance/__init__.py", line 208, in from_config
    return DagsterInstance.from_ref(instance_ref)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/instance/__init__.py", line 216, in from_ref
    run_storage=instance_ref.run_storage,
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/instance/ref.py", line 176, in run_storage
    return self.run_storage_data.rehydrate()
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/serdes/__init__.py", line 222, in rehydrate
    return klass.from_config_value(self, result.value)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/storage/runs/sqlite/sqlite_run_storage.py", line 55, in from_config_value
    return SqliteRunStorage.from_local(inst_data=inst_data, **config_value)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/storage/runs/sqlite/sqlite_run_storage.py", line 60, in from_local
    mkdir_p(base_dir)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/utils/__init__.py", line 124, in mkdir_p
    os.makedirs(path)
  File "/opt/dagster/venv/lib/python3.6/os.py", line 210, in makedirs
    makedirs(head, mode, exist_ok)
  File "/opt/dagster/venv/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/dagster_home'
p
also, can you confirm your dagster version?
a
Copy code
/opt/dagster/app$ pip freeze | grep dagster
WARNING: Could not generate requirement for distribution -agster-graphql 0.7.3 (/opt/dagster/venv/lib/python3.6/site-packages): Parse error at "'-agster-'": Expected W:(abcd...)
dagster==0.7.3
dagster-aws==0.7.3
dagster-cron==0.7.3
dagster-ge==0.7.3
dagster-graphql==0.7.3
dagster-pandas==0.7.3
(venv) ubuntu@myip:/opt/dagster/app$ pip freeze | grep dagit
WARNING: Could not generate requirement for distribution -agster-graphql 0.7.3 (/opt/dagster/venv/lib/python3.6/site-packages): Parse error at "'-agster-'": Expected W:(abcd...)
dagit==0.7.3
but the instance was set up using 0.7.1 I think
p
ah… so this was working and stopped working with the last release?
At first blush, my mind is going to a user permissions error for the $DAGSTER_HOME directory, but if it was working before…
a
No, I hadn't tried using schedules before
this is my first attempt setting them up
p
the error makes me believe that it’s not related to schedules… the error is from the run_storage implementation trying to write to dagster home…
but i guess there’s some path in which the dagster command is not getting things set up correctly
a
I just set up a new env on my local machine, here's what I'm getting now:
p
hang on, I may need to dig into this a bit
a
Copy code
dagster schedule up --preview -y repository.yaml
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: attempt to write a readonly database

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/dagster/cli/__init__.py", line 38, in main
    cli(obj={})  # pylint:disable=E1123
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/dagster/cli/schedule.py", line 133, in schedule_up_command
    return execute_up_command(preview, kwargs, click.echo)
  File "/usr/local/lib/python3.6/dist-packages/dagster/cli/schedule.py", line 143, in execute_up_command
    instance = DagsterInstance.get()
  File "/usr/local/lib/python3.6/dist-packages/dagster/core/instance/__init__.py", line 186, in get
    return DagsterInstance.from_config(_dagster_home())
  File "/usr/local/lib/python3.6/dist-packages/dagster/core/instance/__init__.py", line 208, in from_config
    return DagsterInstance.from_ref(instance_ref)
  File "/usr/local/lib/python3.6/dist-packages/dagster/core/instance/__init__.py", line 216, in from_ref
    run_storage=instance_ref.run_storage,
  File "/usr/local/lib/python3.6/dist-packages/dagster/core/instance/ref.py", line 176, in run_storage
    return self.run_storage_data.rehydrate()
  File "/usr/local/lib/python3.6/dist-packages/dagster/core/serdes/__init__.py", line 222, in rehydrate
    return klass.from_config_value(self, result.value)
  File "/usr/local/lib/python3.6/dist-packages/dagster/core/storage/runs/sqlite/sqlite_run_storage.py", line 55, in from_config_value
    return SqliteRunStorage.from_local(inst_data=inst_data, **config_value)
  File "/usr/local/lib/python3.6/dist-packages/dagster/core/storage/runs/sqlite/sqlite_run_storage.py", line 64, in from_local
    engine.execute('PRAGMA journal_mode=WAL;')
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 2182, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 976, in execute
    return self._execute_text(object_, multiparams, params)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1149, in _execute_text
    parameters,
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context
    e, statement, parameters, cursor, context
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context
    cursor, statement, parameters, context
  File "/usr/local/lib/python3.6/dist-packages/sqlalchemy/engine/default.py", line 588, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database
[SQL: PRAGMA journal_mode=WAL;]
(Background on this error at: <http://sqlalche.me/e/e3q8>)
ok, let me know if I can help
p
What are the permissions on
/dagster_home
?
a
drwxr-xr-x 5 ubuntu ubuntu  4096 Mar  5 15:14 dagster_home
p
huh…. and are there other open dagit / dagster processes that could have hung / crashed in a bad state?
I’d also want to confirm the permissions on
/dagster_home/history/*
a
Copy code
/opt/dagster/dagster_home$ ls -la
total 24
drwxr-xr-x  5 ubuntu ubuntu 4096 Mar  5 15:14 .
drwxr-xr-x  5 ubuntu ubuntu 4096 Feb 25 18:44 ..
-rw-rw-r--  1 ubuntu ubuntu  218 Mar  5 15:14 dagster.yaml
drwxr-xr-x  3 ubuntu ubuntu 4096 Mar  9 22:00 history
drwxr-xr-x  4 ubuntu ubuntu 4096 Mar  9 22:53 schedules
drwxr-xr-x 16 ubuntu ubuntu 4096 Mar  6 20:54 storage
I dont see any dagster or dagit processes in htop
and I just tried running the command again after stoping the dagit service, same result
Copy code
/opt/dagster/dagster_home/history$ ls -la
total 52
drwxr-xr-x 3 ubuntu ubuntu  4096 Mar  9 22:00 .
drwxr-xr-x 5 ubuntu ubuntu  4096 Mar  5 15:14 ..
drwxr-xr-x 2 ubuntu ubuntu  4096 Mar  9 20:41 runs
-rw-r--r-- 1 ubuntu ubuntu 36864 Mar  9 20:40 runs.db
m
can we see
env | grep INSTALL_PATH
on the remote?
a
its not set
m
but
DAGSTER_HOME
is set to
/opt/dagster/dagster_home
as expected?
a
yep
well. its set to
/dagster_home
m
ok, that is definitely the issue
we think we know what is broken in the dagster-aws setup script, but for now, you should be able to edit
~/dagster_home.sh
to fix
a
huh, I dont have that file
p
what about
/etc/profile.d/dagster_home.sh
?
a
yep, its there
here's what it contains:
export DAGSTER_HOME=$INSTALL_PATH/dagster_home
p
yeah, it should be:
export DAGSTER_HOME=/opt/dagster/dagster_home
a
nice, seems like its all set up now
p
thanks for the report…. and your patience debugging!
a
no problem at all! thanks for the swift help again guys!
s
We should definitely have a better error here as well
a
@max @prha Hi guys, I got another issue with the scheduler
my schedules werent running, so I recreated my dagit instance to be safe, but they still didnt work
I tried running the scheduled job .sh file manually
It works if I run in from the
app
folder, but fails otherwise
Copy code
Traceback (most recent call last):
  File "/opt/dagster/venv/bin/dagster-graphql", line 8, in <module>
    sys.exit(main())
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster_graphql/cli.py", line 216, in main
    cli(obj={})  # pylint:disable=E1120
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster_graphql/cli.py", line 208, in ui
    execute_query_from_cli(handle, query, variables, output)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster_graphql/cli.py", line 92, in execute_query_from_cli
    handle, query, variables=seven.json.loads(variables) if variables else None
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster_graphql/cli.py", line 52, in execute_query
    handle=handle, instance=instance, execution_manager=execution_manager, version=__version__
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster_graphql/implementation/context.py", line 18, in __init__
    self.repository_definition = self.get_handle().build_repository_definition()
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/definitions/handle.py", line 489, in build_repository_definition
    obj = self.entrypoint.perform_load()
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/definitions/handle.py", line 546, in entrypoint
    return self.data.get_repository_entrypoint(from_handle=self)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/definitions/handle.py", line 629, in get_repository_entrypoint
    return LoaderEntrypoint.from_yaml(self.repository_yaml, from_handle=from_handle)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/definitions/handle.py", line 246, in from_yaml
    return LoaderEntrypoint.from_module_target(module_name, fn_name, from_handle)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/definitions/handle.py", line 232, in from_module_target
    module = importlib.import_module(module_name)
  File "/opt/dagster/venv/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'repository'
p
@sashank ^^ any ideas?
s
I think it might be a similar problem to this: https://github.com/dagster-io/dagster/issues/2185
a
well, since my executions are all gonna be run in the dagit instance, just adding a simple
cd /opt/dagster/app
to the generated scripts should do the trick, yes?
s
Let me try to repro this and get a clear answer to you
a
fyi, it worked, but I have to edit the .sh files every time I deploy my repo with
dagster-aws up
n
@Auster Cid thanks again for flagging, this is now fixed on master: https://dagster.phacility.com/D2304
a
cool! thanks for letting me know! btw, for anyone stumbling into this chat in the future, my solution here:
well, since my executions are all gonna be run in the dagit instance, just adding a simple
cd /opt/dagster/app
to the generated scripts should do the trick, yes?
is bad and you should install your app as an editable package like in the thread sashank linked.