Channels
dagster-de
topic-async-execution
dagster-jp
dagster-cube
gigs-freelance
dagster-wandb
dagster-feedback
dagster-machine-learning
dagster-bigquery
dagster-support
dagster-dask
dagster-snowflake
dagster-releases
dagster-dbt
faq-read-me-before-posting
tools
dagster-ext
project-flexible-range-backfills
data-platform-design
events
dagster-serverless
dagster-noteable
introductions
dagster-airflow
dagster-cloud
random
dagster-airbyte
dagster-spatial
announcements
github-discussions
豆瓣酱帮
dagstereo
dagster-ecs
dagster-kubernetes
jobs
dagster-showcase
data-quality-asset-checks
Powered by
#announcements
Title
e
eamag
09/27/2019, 11:59 AM
hey everyone! Quick question: how can I reuse output of the solid, should I just put it twice in the pipeline? Will it be calculated twice or not?
So I want to download and transform the data in one solid, then split pipeline in branches, calculate in parallel and combine the output and write it
p
Philipp G
09/27/2019, 1:12 PM
Sounds like the example that is explained in the tutorial
https://dagster.readthedocs.io/en/0.5.9/sections/learn/tutorial/actual_dag.html
Parallel computing is not a part of this small example, but you will learn how to produce multiple outputs and how to combine them again.
e
eamag
09/27/2019, 1:17 PM
oh thanks, missed multiple times somehow 😅
👍 1
m
max
09/27/2019, 4:15 PM
you can have multiple solids depend on the same output 🙂
it won't be calculated twice
Post