https://dagster.io/ logo
#announcements
Title
# announcements
s

SamSinayoko

06/01/2020, 7:16 AM
Hi, I've been struggling to get the stocks backfill example running:
Copy code
(dagster) $ DAGSTER_HOME=$PWD dagster pipeline backfill  -y repository.yaml
Select a pipeline to backfill: compute_total_stock_volume: compute_total_stock_volume

     Pipeline: compute_total_stock_volume
Partition set: stock_data_partitions_set
   Partitions: 
                2018-01-01           2018-02-01           2018-03-01           2018-04-01           2018-05-01           2018-06-01           2018-07-01           2018-08-01
                2018-09-01           2018-10-01           2018-11-01           2018-12-01

Do you want to proceed with the backfill (12 partitions)? [y/N]: y
Launching runs... 
Traceback (most recent call last):
  File "/Users/sinayoks/apps/miniconda3/envs/dagster/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/dagster/cli/__init__.py", line 38, in main
    cli(obj={})  # pylint:disable=E1123
  File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/dagster/cli/pipeline.py", line 599, in pipeline_backfill_command
    execute_backfill_command(kwargs, click.echo)
  File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/dagster/cli/pipeline.py", line 693, in execute_backfill_command
    instance.launch_run(run.run_id)
  File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/dagster/core/instance/__init__.py", line 801, in launch_run
    return self._run_launcher.launch_run(self, run)
  File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/dagster_graphql/launcher/__init__.py", line 97, in launch_run
    cls=self.__class__.__name__, address=self._address, result=result
dagster.core.errors.DagsterLaunchFailedError: Failed to launch run with RemoteDagitRunLauncher targeting <http://127.0.0.1:3000>:
{'__typename': 'PipelineRunNotFoundError'}
So it looks like the job is creating a run but Dagit isn't able to find it. Any idea what I'm doing wrong here?
a

alex

06/01/2020, 1:37 PM
It seems that the
dagit
you have running was not booted with the same
repository.yaml
edit: I misread this thought it was
PipelineNotFoundError
- as max comments below you need to make sure you are using the same
DagsterInstance
in both which can be done by setting
DAGSTER_HOME
env var
s

SamSinayoko

06/01/2020, 3:40 PM
I don't think it's that though, I'm pointing to the same repository.yaml in the attached screenshot and am getting the same error.
Have only got the following in dagster.yaml
Copy code
run_launcher:
  module: dagster_graphql.launcher
  class: RemoteDagitRunLauncher
  config:
    address: "<http://127.0.0.1:3000>"
and this in repository.yaml
Copy code
repository:
  module: repository
  fn: define_repo
  pipeline: compute_total_stock_volume
a

alex

06/01/2020, 4:03 PM
what do you see when you open dagit?
m

max

06/01/2020, 4:51 PM
what is the
DAGIT_HOME
env variable
👍 1
s

SamSinayoko

06/01/2020, 5:24 PM
DAGIT_HOME is empty.
m

max

06/01/2020, 5:25 PM
cool
a

alex

06/01/2020, 5:26 PM
m

max

06/01/2020, 5:26 PM
i think in one of the above screenshots you are setting
DAGIT_HOME
in one incantation and
DAGSTER_HOME
in another
you should only set
DAGSTER_HOME
👍 1
s

SamSinayoko

06/01/2020, 5:28 PM
🤦‍♂️ that was it, works now thanks!
🎊 1
Is there a way to trigger a backfill from dagit itself? I can see how to trigger individual partitions but don't know if there's a way to say "run all". Does that have to go in a preset?
a

alex

06/01/2020, 5:34 PM
no “run all” from dagit at this time
s

SamSinayoko

06/01/2020, 5:35 PM
👍