Hey all I have been pleasantly surprised at the level of abs

Question

Hey all I have been pleasantly surprised at the level of abstractions available in dagster I really like it the main reason being that it embraces declarative and compositional patterns Btw I have been an early user of graphql since 2015 and that could be a reason why I like dagster too I was wondering if there is an architectural view of dagster somewhere What I am looking for is how the core packages tie together I am pretty familiar with the JS ecosystem I took a quick look at the repo and found dagit to be the react front end made with create react app apollo etc Dagster is the python module what s scala used for The graphQL server is within dagster or is there a separate server component I am planning to use dagster for my startup my use case would be different from the core ML amp ETL use cases dagster is designed for So looking at the architecture I could probably customize my deployments

Jai Kumaran · Answer

Ok so I took a peek into the python modules dagit And we seem to be using flask as the webserver and there is a flask graphql too I really do wish there was a high level picture which shows all this

Jai Kumaran · Answer

Looking at more of the code I am really liking what you guys are doing here Would love to hear a talk on your design decisions for things like grpc

Noah K · Answer

Without having looked I would assume the scala code is to integrate with Spark

nate · Answer

hey Yes the Scala code is just a demo for Scala Spark In terms of organization otherwise you ll find all of Dagster core code under `python modules dagster` integrations with other systems under `python modules libraries` and as you ve seen the Dagit backend is under `python modules dagit`

Jai Kumaran · Answer

< nate> thanks much This is great