@sandy It would be amazing for you to help us get started!
I’m also at an org with Airflow experience, where we’ve added custom improvements on top of Airflow to deal with its flaws. One flaw that we’re never going to work around is the ability to run workflows locally. I want to run a thing on my macbook, then have the identical thing run in AWS (or GCP). This is the first “win” that Dagster could offer. To that end, I’ve got a simple workflow I’m running on my Mac. I now need to figure out “modes” such that I can represent “environments”, ie. “dev” (local) vs “test” (AWS account 1) vs “prod” (AWS account 2). The differences in environment must account for IO/storage (local filesystem on a mac vs. S3 bucket in cloud). My simple workflow has a step to touch a file to create an Asset to illustrate this. I think this will amount to the creation of a
dagster.yaml
and figuring out the steps to go through dev -> test -> prod. Any illustrative examples of this would be greatly appreciated.