https://dagster.io/ logo
#announcements
Title
# announcements
w

Wali

09/23/2020, 6:43 PM
Can anyone help me to implement dagster with Azure databricks and ADLS gen2?
m

max

09/23/2020, 6:52 PM
hi wali
you'll probably also want to take a look at dagster_databricks https://github.com/dagster-io/dagster/tree/master/python_modules/libraries/dagster-databricks
w

Wali

09/23/2020, 7:39 PM
@max thanks. I was sort of looking for a step by step tutorial. I am familiar with implementation of Great Expectations both locally and on azure databricks. For dagster, do I need to implement it locally first and then copy the project folder onto Databricks DBFS? How do I allow multiple teams to collaborate on the same pipeline if I implement it locally?
m

max

09/23/2020, 7:56 PM
hm, i can probably help with particular tasks but i'm not quite sure what exactly you're aiming at
what's your desired end state?
w

Wali

09/23/2020, 8:00 PM
Implement a POC pipeline using Azure Databricks and ADLSgen2. Then multiple data engineers should be able to update the pipeline following a process.
m

max

09/23/2020, 9:31 PM
@sandy do you know if the spark solids will just work on azure
w

Wali

09/25/2020, 6:34 PM
@max @sandy any update on this? would you be able to help?
m

max

09/28/2020, 2:16 PM
hi wali, it's still a little hard to know what exactly you need
i don't think either of us is going to be able to write a tutorial for running spark solids on azure in the next couple of days
but if you have a pipeline that works locally and you run into specific issues getting it working with azure/adls2, we can certainly advise on that
s

sandy

09/28/2020, 3:09 PM
wali and I are going to chat today
w

Wali

09/28/2020, 4:41 PM
👍
5 Views