Hi everyone, I'm new to dagster and actually orchestration in general! One of the things that drew me to dagster was the separation of inputs and outputs, in particular with different configured environments (if that's the correct term), i.e. local, development and production. I'm however struggling to set this up. From the documentation I believe it depends on assets, IO managers and resources but could someone please explain further these concepts and interdependencies or point me towards a simple tutorial? Any help is much appreciated! Thank you!
Hi Sandy, thank you for the links! I'm afraid I still don't quite understand though. Maybe it's best if I explain this experimental project I'm working on? It consists of pulling Spotify tracks via their api, extracting the artists from the songs and comparing them to concert venues to find potential gigs, scheduled weekly.
Therefore the "prod" version of this I'd want to run multiple api calls with pagination. Then do some data extraction to get all the unique artists to compare to venue gigs. With the "local" version for testing I'd ideally just load in a csv of the songs.
• My understanding is that because Spotify would be a API call, then it should be a Resource? Which is then utilised within an Asset?
◦ But then also because it requires multiple steps in turning the api call into a dataframe, maybe it should be a "Graph-backed asset" in order to allow for separation of steps via "ops"?
• The Asset that I defined never seemed to show up in dagit, so I'm assuming I'm doing something wrong with the way it is setup?
• Still not sure how the separation of environments work. Even if I did manage to get "local_assets" and "prod_assets" to work, where is it that you define which one should run within the graph?
Hopefully that all makes sense and that the files can assist in explaining where I'm going wrong, to make up for my lack of technical knowledge/language.
Thank you again for your help!
Hi @sandy, I realised I didn't @ you, so you might not have seen this? (Also no problem at all if you have and just haven't had time to reply!)
07/12/2022, 3:56 PM
Hey Lawrence, I wrote up how I would do this if I were in your position: https://github.com/dagster-io/dagster/discussions/8845. I made it a github discussion so people coming from the internet with similar questions can find it. Does the implementation there make sense?
07/12/2022, 8:12 PM
Hi @sandy thank you for doing this! I really appreciate it!