Hey all, I have been pleasantly surprised at the l...
# announcements
j
Hey all, I have been pleasantly surprised at the level of abstractions available in dagster. I really like it, the main reason being that it embraces declarative and compositional patterns. Btw, I have been an early user of graphql since 2015 and that could be a reason why I like dagster too. I was wondering if there is an architectural view of dagster somewhere? What I am looking for is how the core packages tie together. I am pretty familiar with the JS ecosystem. I took a quick look at the repo and found dagit to be the react front end, made with create-react-app/apollo/etc. Dagster is the python module (?) what’s scala used for ? The graphQL server is within dagster or is there a separate server component? I am planning to use dagster for my startup, my use case would be different from the core (ML & ETL) use cases dagster is designed for. So looking at the architecture, I could probably customize my deployments.
Ok, so I took a peek into the python_modules/dagit. And we seem to be using flask as the webserver and there is a flask-graphql too! I really do wish there was a high level picture which shows all this.
Looking at more of the code. I am really liking what you guys are doing here. Would love to hear a talk on your design decisions for things like grpc
n
Without having looked, I would assume the scala code is to integrate with Spark
👍 1
n
hey! Yes, the Scala code is just a demo for Scala Spark. In terms of organization otherwise, you’ll find all of Dagster core code under
python_modules/dagster
, integrations with other systems under
python_modules/libraries
, and as you’ve seen the Dagit backend is under
python_modules/dagit
👍 1
j
@nate thanks much. This is great!