https://dagster.io/ logo
#announcements
Title
# announcements
z

Zach

04/10/2020, 3:25 PM
This is probably a dumb question. Is it possible (or advisable) to make full use of dagster without setting up configs in yaml files? I'm not sure if I understand the benefit entirely, and it feels like it adds extra layers of complication. I guess I also don't like yaml's syntax, so that's probably some bias on my part. Thanks in advance for helping with my ignorance. 🙂
s

schrockn

04/10/2020, 3:29 PM
@Zach  yes that is possible. all the python APIs operate on python dictionaries (and the config typing applies to the python dictionary) so the system can be used at that level without level
the config editor in the playground is deeply tied to yaml so you would not be able to use that
z

Zach

04/10/2020, 3:34 PM
Cool. Thanks for your work on this, btw @schrockn. I'm not a professional data engineer but I'm starting to see some of the brilliance in the design of Dagster. I worry I don't have a full understanding of the advantages of the config editor, and maybe I'm incorrectly avoiding it. We're a super small team switching from a crappy SSIS ETL process to something less painful and also something that finally lets us have a 'real' ML pipeline for some of that work.
s

schrockn

04/10/2020, 3:35 PM
Thanks for the kind words!
The advantage of the config editor is schema and autocompletion over your configuration documents.
What do you use for configuration right now?
z

Zach

04/10/2020, 3:40 PM
Still in super early POC phase, and probably using it incorrectly in some way. I'm just using the configuration aspect for handling the database keys/hosts between dev and prod, in the resource object. The actual SQL for the solids is just plopped into the python code for the solids themselves.
I've not begun to actually build any solids for orchestration our models. But I did see some good examples you guys made on that part.
s

schrockn

04/10/2020, 3:41 PM
I guess my question is that if you weren't using dagster and still wanted to provide db keys/hosts as config what would you use?
z

Zach

04/10/2020, 3:44 PM
Hah, no idea what the best practice is for that. I'm not sure what I would do. That's how green I am, coming into this. I'm a data scientist that's fed up with our current rubberband+chewing gum setup - this isn't my forte.
s

schrockn

04/10/2020, 3:44 PM
got it
yeah if you've never with the hell of totally ad-hoc json config or something similar the config system will seem like overkill
the config system provides documentation, good error messages, and a nice editor for configuration documents
z

Zach

04/10/2020, 3:46 PM
I'm guessing the need will become more apparent as this gets more complex. Thanks for your advice
s

schrockn

04/10/2020, 3:46 PM
👍