Hey guys good day! Unlike Airflow, is that possible to use Dagster as a pure library for a Flask app without having to use CLI to trigger tasks? Reason asking is that our company’s hosts are having problems to install the Airflow dependencies and most importantly, Airflow requires to run as a standalone app which makes things complicated since it cannot easily blend into our deployment environment.

Hi Xu - we do have support for this, yes. There's an execute_pipeline function call that you can use for this if you have a PipelineDefinition and want to execute it on the same machine making the function call: https://docs.dagster.io/_apidocs/execution#dagster.execute_pipeline Or if you want to be able to use a run launcher to execute your pipeline in a different process like dagit does, we have an option for that as well, it's just a little bit more hidden currently

Nice! Thank you Daniel! Other questions: 1. Can it be configured to use the same MySQL DB that the Flask uses to bootstrap schemas that it needs to manage states of tasks? 2. If the answer to the first question is yes, is that possible to run multiple Flask + Dagster instances accessing the same MySQL DB without creating race conditions? 3. Is that possible to add the views of nice Dagit UI into the Flask? I am trying to use awesome features provided by Dagster without having to spin up another process...

This all sounds doable - one small blocker is that we don't currently have a mysql storage plugin, but that would be quite easy to add (we use sqlalchemy and have plugins for sqlite and postgres)

Nice! My Flask app uses SQLAlchemy as well, is not SQLAlchemy DB agnostic?

yeah, we would need to write a class like this for MySQL that configures sqlalchemy to use postgres, most of the implementation would already be handled: https://dagster.phacility.com/source/dagster/browse/master/python_modules/libraries/dagster-postgres/dagster_postgres/run_storage/run_storage.py$10

For the UI part, as long as I can view the task status it should be fine.. is Dagit running on Flask?

It is!

That’s awesome, then I guess it will make hacking bit easier ;)

I don't think that's currently part of our roadmap for the next release. It's something we'd be able to help out with with pointers though, and we can put it in the triage list for future releases as well

thank you!

I just talked this over with @alex a bit and doing this may be a little more involved than I thought - there are just a few edge cases we'd need to figure out, like how to subscribe to changes in a mysql-based event log. It may be that this is less of just a small tweak than I previously described :/

got it, no problem! Thanks again!

I will still put it in the triage list though! I think this would make sense as a project for our 0.11.0 release, so not the one coming out in a few weeks but the one after that

Thank you so much! I’m looking forward to it! This is exciting!

right now we're all pretty focused on our 0.10.0 release coming out in mid-December. This feels feasible to do after that, so roughly in the January timeframe seems possible? I'd have to check with the rest of the team though

Got it thanks

Hi Xu - the 'roughly in the January timeframe' estimate still feels about right, this seems like something we can do shortly after 0.10.0 is released

thank you!