Hey guys good day! Unlike Airflow, is that possibl...
# announcements
x
Hey guys good day! Unlike Airflow, is that possible to use Dagster as a pure library for a Flask app without having to use CLI to trigger tasks? Reason asking is that our company’s hosts are having problems to install the Airflow dependencies and most importantly, Airflow requires to run as a standalone app which makes things complicated since it cannot easily blend into our deployment environment.
d
Hi Xu - we do have support for this, yes. There's an execute_pipeline function call that you can use for this if you have a PipelineDefinition and want to execute it on the same machine making the function call: https://docs.dagster.io/_apidocs/execution#dagster.execute_pipeline Or if you want to be able to use a run launcher to execute your pipeline in a different process like dagit does, we have an option for that as well, it's just a little bit more hidden currently
x
Nice! Thank you Daniel! Other questions: 1. Can it be configured to use the same MySQL DB that the Flask uses to bootstrap schemas that it needs to manage states of tasks? 2. If the answer to the first question is yes, is that possible to run multiple Flask + Dagster instances accessing the same MySQL DB without creating race conditions? 3. Is that possible to add the views of nice Dagit UI into the Flask? I am trying to use awesome features provided by Dagster without having to spin up another process...
d
This all sounds doable - one small blocker is that we don't currently have a mysql storage plugin, but that would be quite easy to add (we use sqlalchemy and have plugins for sqlite and postgres)
x
Nice! My Flask app uses SQLAlchemy as well, is not SQLAlchemy DB agnostic?
And for things doable, are they doable today with straight documentation or do I need to do a little bit of hacking?
d
yeah, we would need to write a class like this for MySQL that configures sqlalchemy to use postgres, most of the implementation would already be handled: https://dagster.phacility.com/source/dagster/browse/master/python_modules/libraries/dagster-postgres/dagster_postgres/run_storage/run_storage.py$10
That's probably the major bit of hacking that would need to happen - I'm not entirely sure about the dagit UI part, I think it depends on what exactly you're hoping to bring in
x
For the UI part, as long as I can view the task status it should be fine.. is Dagit running on Flask?
d
It is!
x
That’s awesome, then I guess it will make hacking bit easier ;)
@daniel for the plugin for MySQL, is there any plan to make it happen?
d
I don't think that's currently part of our roadmap for the next release. It's something we'd be able to help out with with pointers though, and we can put it in the triage list for future releases as well
x
thank you!
morning Daniel, after watching and reading tech talks and docs, I really like the idea of Dagster. The only blocker for me is that our company only has MySQL cluster as our DB. I understand it’s not in the current planning, so I want to doing some local hacking to support MySQL, but if you could kindly provide me some guidance or even a gist, that would be really helpful!
@daniel sorry to ping, just wanna make sure you don’t miss the message above as it is an old thread
d
I just talked this over with @alex a bit and doing this may be a little more involved than I thought - there are just a few edge cases we'd need to figure out, like how to subscribe to changes in a mysql-based event log. It may be that this is less of just a small tweak than I previously described :/
x
got it, no problem! Thanks again!
d
I will still put it in the triage list though! I think this would make sense as a project for our 0.11.0 release, so not the one coming out in a few weeks but the one after that
x
Thank you so much! I’m looking forward to it! This is exciting!
In the meantime I guess I will just use SQLite and obviously I can’t not convince people with the concept of Dagster with no-MySQL support as that’s the only relational DB we support
@daniel may I have a rouge ETA to support MySQL?
d
right now we're all pretty focused on our 0.10.0 release coming out in mid-December. This feels feasible to do after that, so roughly in the January timeframe seems possible? I'd have to check with the rest of the team though
x
Got it thanks
@daniel I’ve demoed the dagster with our core team and they liked it a lot, the only blocker would be the MySQL support. Would you mind to provide a rough ETA about MySQL support?
d
Hi Xu - the 'roughly in the January timeframe' estimate still feels about right, this seems like something we can do shortly after 0.10.0 is released
x
thank you!