welcome!

added an integration to this channel: Release Bot

Hello, world!

Just out here in robot land glorying in our successful release of 0.4.3... a limited existence tbh.

Max Gasner just published a new version: <click.core.Command object at 0x10f5ff0b8>. Don't forget to switch the active version of the docs at ReadTheDocs!

hmmmm

Max Gasner just published a new version: 0.4.3.post4. Don't forget to switch the active version of the docs at ReadTheDocs!

Hey everyone, haven’t poked around here in a while. I’m looking at how dagster may replace one of my typical pipeline use cases. Running a pipeline via airflow or luigi, where airflow/luigi are in local execution mode running as a service in a single container, and jobs are run as containers on ECS/Batch/Kubernetes. Pipelines can take minutes to hours to run, so tracking pipeline state and being able to recover is important. Airflow and luigi handle this by tracking individual task and overall workflow state, saving to a database or disk. In Dagster world, is there a way to track and recover pipelines from a state? Looking through the code, I see ReexecutionConfig, but not sure how to get it or construct it. Maybe by tracking execution through execute_pipeline_iterator and recording the StepOutputHandles?

the easiest is through dagit

*easiest way

so configure the pipeline to persist intermediates to disk

(or to s3 if you are executing in a cluster)

via

storage:
   filesystem:

in a config file

if you then execute dagit

in the run UI

there are arrows next to the execution steps in the righthand pane

if you need the python API, we can discuss that too

I may be missing something. I mean automatic, like the container running dagster dies 45 minutes into a 3 hour pipeline and a few steps have completed. When the container restarts, picking up where it left off.

we do not have any explicit support for that kind of retry logic atm

however that is a logical extension of the current infrastructure

we prefer to think of it as incremental computation

so you would just rerun the dag and the steps would be skipped that have already been successfully computed

Like individual nodes can retry, and even the container services offer basic retries, but dagster is the only thing that knows the overall workflow / high level task.

ya

Ah, as in, rerun the nodes, but they somehow detect they’ve run, using say a datetime or run ID?

runID

as of now

the caller is responsible for threading through the correct previous run ID

and doing a fair amount of orchestration

That actually may work well, since dagster could be run by a job system with retries itself, that tracks the run ID and other params. Dagster being the “head” job