Ivan Doroshenko

01/10/2022, 10:25 AM
Hi! I’m curious is it pragmatic to use Dagster as tool of choice to orchestrate mixed purpose workflows. Initial idea is to introduce gradual automatisation of business tasks (tasks that very loosely related with creation of data assets) by implementing workflow engine (aka runbooks) alongside with data pipeline using same tooling. Proprietary can be example of such tool. So, how convenient is to use Dagster as general purpose workflow engine? Is it against Dagster philosophy?


01/10/2022, 5:36 PM
Whilst not common; it certainly is possible - see
❤️ 1
You could also use sensors to detect some external condition (eg; a new file in a S3 bucket or some API response) and trigger new job runs.