The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

This is probably a dumb question. Is it possible (or advisable) to make full use of dagster without setting up configs in yaml files? I'm not sure if I understand the benefit entirely, and it feels like it adds extra layers of complication. I guess I also don't like yaml's syntax, so that's probably some bias on my part. Thanks in advance for helping with my ignorance. :slightly_smiling_face:

<@U011ALQCX61>  yes that is possible. all the python APIs operate on python dictionaries (and the config typing applies to the python dictionary) so the system can be used at that level without level

the config editor in the playground is deeply tied to yaml so you would not be able to use that

Cool. Thanks for your work on this, btw <@UCCLPK5G8>. I'm not a professional data engineer but I'm starting to see some of the brilliance in the design of Dagster. I worry I don't have a full understanding of the advantages of the config editor, and maybe I'm incorrectly avoiding it. We're a super small team switching from a crappy SSIS ETL process to something less painful and also something that finally lets us have a 'real' ML pipeline for some of that work.

The advantage of the config editor is schema and autocompletion over your configuration documents.

What do you use for configuration right now?

Still in super early POC phase, and probably using it incorrectly in some way. I'm just using the configuration aspect for handling the database keys/hosts between dev and prod, in the resource object. The actual SQL for the solids is just plopped into the python code for the solids themselves.

I've not begun to actually build any solids for orchestration our models. But I did see some good examples you guys made on that part.

I guess my question is that if you weren't using dagster and still wanted to provide db keys/hosts as config what would you use?

Hah, no idea what the best practice is for that. I'm not sure what I would do. That's how green I am, coming into this. I'm a data scientist that's fed up with our current rubberband+chewing gum setup - this isn't my forte.

yeah if you've never with the hell of totally ad-hoc json config or something similar the config system will seem like overkill

the config system provides documentation, good error messages, and a nice editor for configuration documents

I'm guessing the need will become more apparent as this gets more complex. Thanks for your advice