Hello everyone, is there any beginners channel where I can ask some potentially dumb questions about getting started with Dagster?
02/17/2023, 5:43 PM
hi @Pragun Bhutani -- this is the right channel for questions of any level 🙂
02/20/2023, 9:25 AM
Thank you @owen!
I'm currently working on a project that's something like a Zapier clone - where my users are able to program simple data transfer or transformation jobs for themselves and set them to a schedule.
For my backend, I currently use Airflow with a custom DAG CRUD API which copy-pastes templated DAG files into my DAG folder every time I want to add a new job to the schedule / or deletes then when I want to remove my job etc.
However, in the interest of running less software on my own and saving some time, I was wondering it was possible to achieve the same functionality using Dagster running on Dagster cloud?
02/21/2023, 6:58 PM
Ah that's an interesting question! It sounds like you already have the trickiest part figured out (a way to template out code artifacts and get them into a central location). One difference between Dagster and Airflow's management of code locations is that in the UI, Airflow continually re-reads the main dag folder every few seconds (so any changes will be automatically picked up), whereas Dagster will only re-load it's interpretation of the code state on request (i.e. there's a "reload code location" button that you can click). This can also be automated via the GraphQL API so I don't think this would be a blocker by any means, just something to look out for