The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

If I have a large pipeline DAG is there a way I can locally develop on a particular asset using input assets from production?

For example my pipeline contains an initial step of data extraction from another system. This step takes a long time whether run locally or otherwise, so I'd like to not have to repeat it locally.

Ideally I'd be able to download those assets from production and then run the rest of the pipeline locally to develop the pipeline tasks.

Is this something which Dagster supports?

(assume the pipeline has already anonymised etc. the data before the step I'm talking about)

hey <@U05H67WFGAK> there’s an experimental feature that can help with this. I wrote a github discussion about it here <https://github.com/dagster-io/dagster/discussions/16145>