@max thanks. I was sort of looking for a step by step tutorial. I am familiar with implementation of Great Expectations both locally and on azure databricks. For dagster, do I need to implement it locally first and then copy the project folder onto Databricks DBFS? How do I allow multiple teams to collaborate on the same pipeline if I implement it locally?
09/23/2020, 7:56 PM
hm, i can probably help with particular tasks but i'm not quite sure what exactly you're aiming at
what's your desired end state?
09/23/2020, 8:00 PM
Implement a POC pipeline using Azure Databricks and ADLSgen2. Then multiple data engineers should be able to update the pipeline following a process.
09/23/2020, 9:31 PM
@sandy do you know if the spark solids will just work on azure
09/25/2020, 6:34 PM
@max@sandy any update on this? would you be able to help?
09/28/2020, 2:16 PM
hi wali, it's still a little hard to know what exactly you need
i don't think either of us is going to be able to write a tutorial for running spark solids on azure in the next couple of days
but if you have a pipeline that works locally and you run into specific issues getting it working with azure/adls2, we can certainly advise on that