SamSinayoko
06/01/2020, 7:16 AM(dagster) $ DAGSTER_HOME=$PWD dagster pipeline backfill -y repository.yaml
Select a pipeline to backfill: compute_total_stock_volume: compute_total_stock_volume
Pipeline: compute_total_stock_volume
Partition set: stock_data_partitions_set
Partitions:
2018-01-01 2018-02-01 2018-03-01 2018-04-01 2018-05-01 2018-06-01 2018-07-01 2018-08-01
2018-09-01 2018-10-01 2018-11-01 2018-12-01
Do you want to proceed with the backfill (12 partitions)? [y/N]: y
Launching runs...
Traceback (most recent call last):
File "/Users/sinayoks/apps/miniconda3/envs/dagster/bin/dagster", line 8, in <module>
sys.exit(main())
File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/dagster/cli/__init__.py", line 38, in main
cli(obj={}) # pylint:disable=E1123
File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/dagster/cli/pipeline.py", line 599, in pipeline_backfill_command
execute_backfill_command(kwargs, click.echo)
File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/dagster/cli/pipeline.py", line 693, in execute_backfill_command
instance.launch_run(run.run_id)
File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/dagster/core/instance/__init__.py", line 801, in launch_run
return self._run_launcher.launch_run(self, run)
File "/Users/sinayoks/apps/miniconda3/envs/dagster/lib/python3.6/site-packages/dagster_graphql/launcher/__init__.py", line 97, in launch_run
cls=self.__class__.__name__, address=self._address, result=result
dagster.core.errors.DagsterLaunchFailedError: Failed to launch run with RemoteDagitRunLauncher targeting <http://127.0.0.1:3000>:
{'__typename': 'PipelineRunNotFoundError'}
So it looks like the job is creating a run but Dagit isn't able to find it. Any idea what I'm doing wrong here?Ryan Tam
06/01/2020, 7:29 PMMuthu
06/01/2020, 7:59 PM/
…
i want to run the dagit like: <http://localhost:3000/engine>
Binh Pham
06/01/2020, 8:00 PMTravis Cline
06/01/2020, 10:07 PMTravis Cline
06/01/2020, 10:12 PMTravis Cline
06/01/2020, 10:12 PMBinh Pham
06/02/2020, 11:39 PMsephi
06/03/2020, 6:53 AM@composite_solid(
output_defs=[OutputDefinition(name='path', dagster_type=String),
OutputDefinition(name='dataframe', dagster_type=DataFrame),
])
def my_solid():
...
path = solid_return_path()
dataframe = solid_return_dataframe()
yield Output (path, 'path')
yield Output (dataframe, 'dataframe')we get the following error:
dagster.core.errors.DagsterInvalidDefinitionError: @composite_solid my_solid returned problematic value of type <class 'generator'>. Expected return value from invoked solid or dict mapping output name to return values from invoked solids.If we change it to :
return path, dataframethe pipeline runs. Any suggestions?
Sam Rausser
06/04/2020, 2:22 AMSam Rausser
06/04/2020, 2:23 AMSam Rausser
06/04/2020, 2:23 AMmax
06/04/2020, 2:28 AManderson
06/04/2020, 1:30 PMSam Rausser
06/04/2020, 2:38 PMSam Rausser
06/04/2020, 2:39 PMcompute_logs:
module: dagster.core.storage.local_compute_log_manager
class: NoOpComputeLogManager
config:
base_dir: /tmp/datafarm
user
06/04/2020, 11:33 PMprha
06/04/2020, 11:40 PMSam Rausser
06/05/2020, 12:23 AM0.7.16
Sam Rausser
06/05/2020, 12:23 AMSam Rausser
06/05/2020, 12:26 AMSam Rausser
06/05/2020, 12:26 AMSam Rausser
06/05/2020, 1:35 AMSam Rausser
06/05/2020, 1:35 AMSam Rausser
06/05/2020, 1:36 AMAndrey Alekseev
06/05/2020, 5:19 PMMuthu
06/05/2020, 7:45 PMsqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedColumn) column "snapshot_id" of relation "runs" does not exist
LINE 1: INSERT INTO runs (run_id, snapshot_id, pipeline_name, status...
VK
06/06/2020, 6:55 AMVK
06/06/2020, 6:56 AMwbonelli
06/06/2020, 7:22 PMexecute_pipeline_iterator
to send logs to the event stream?wbonelli
06/06/2020, 7:22 PMexecute_pipeline_iterator
to send logs to the event stream?schrockn
06/06/2020, 7:25 PMwbonelli
06/06/2020, 7:46 PMalex
06/06/2020, 7:59 PM