has anyone ever tried running databricks connect in their us

Question

has anyone ever tried running databricks connect in their user code to run spark code within dagster ops on a remote databricks cluster

Huib Keemink · Answer

I have It works but you lose some nice properties like expectations on the spark dataframes I think it really depends on what your compute looks like if this solution makes sense if you just need to do some filtering aggregation in spark to produce a pandas or similar dataframe and this dataframe is small enough to fit in memory on the dagster workers it works quite nicely

Huib Keemink · Answer

For me the machines that dagster lives on are too small to do our data processing so I ended up doing everything on databricks

Zach · Answer

hey thanks for your feedback I m in a similar boat I don t plan on deploying dagster on spark clusters that are the size I d need but would love to have at least some of the data processing logic surfaced in ops even if it s running through databricks connect or even just be able to use dbfs so I can do stuff like gather delta table metadata to report on as AssetMaterializations Observations Sounds like it s worth giving a shot

Huib Keemink · Answer

Yeah both options are quite nice IMO With the dagster databricks integration you gain the possibility to chain ops that fully live within databricks which makes working with larger datasets super nice because you can just pass a spark dataframe around With the databricks connect flow your setup is a bit easier and you don t risk spinning up a 250GB ram cluster for a simple pandas operation slightly smiling face

Zach · Answer

To clarify the dagster databricks integration are you using the `databricks pyspark step launcher ` and this still doesn t let you pass DataFrames between ops does it

Huib Keemink · Answer

Correct I use the `databricks pyspark step launcher` together with an IO manager that persists dataframes as parquet files

Huib Keemink · Answer

it s a bit of a leaky abstraction because you lose some information in the process but if you re aware of this conversion between ops it s not so bad

Zach · Answer

very cool I ll have to try that out I suppose you can t have different ops in the same job use different launchers can you run launchers seem to be global to each user code deployment

Huib Keemink · Answer

oh no you can

Huib Keemink · Answer

``` op required resource keys= schedule databricks def write datalake context df pass```

Huib Keemink · Answer

for instance this would run on databricks given that the databricks resource is a configured run launcher

Zach · Answer

ahh wunderbar

Huib Keemink · Answer

so my flow is dagster k8s gt job k8s gt databricks job

Huib Keemink · Answer

dagster kicks off a kubernetes job which basically zips itself and uploads the code to databricks which runs the code the k8s job then pulls that result from databricks and gives it back to the dagit UI

Huib Keemink · Answer

for me the biggest benefit is that I don t need databricks for everything sending a slack notification is 100 fine within k8s

Huib Keemink · Answer

however keep in mind that if you plan to share data between databricks ops and k8s ops that both need to be able to access it So an IO manager that stores data in dbfs doesn t really work S3 would

Zach · Answer

yeah that s basically where I m trying to get my flow is starting to look very similar dagster ECS gt job ECS gt databricks job although we ve been using a custom op that uses the jobs run now endpoint to launch predefined notebook jobs as my teammates are still pretty reliant on notebooks for getting logging info from the jobs and most of our pipeline code is in notebooks as of right now crying cat face it appears the databricks pyspark step launcher tries to solve this by reading the logs from S3 after the run has completed do you find that to be pretty reliable

Huib Keemink · Answer

Getting the logs has been underwhelming in my experience I typically end up with a generic databricks error when something went wront

Huib Keemink · Answer

```dagster databricks databricks DatabricksError Run 246292 failed with result state DatabricksRunResultState Failed Message File usr local lib python3 8 site packages dagster core execution api py line 775 in pipeline execution iterator for event in pipeline context executor execute pipeline context execution plan File usr local lib python3 8 site packages dagster core executor in process py line 38 in execute yield from iter File usr local lib python3 8 site packages dagster core execution api py line 869 in iter yield from self iterator File usr local lib python3 8 site packages dagster core execution plan execute plan py line 86 in inner plan execution iterator for step event in check generator dagster event sequence for step step context File usr local lib python3 8 site packages dagster core execution plan execute plan py line 321 in dagster event sequence for step raise unexpected exception File usr local lib python3 8 site packages dagster core execution plan execute plan py line 222 in dagster event sequence for step for step event in check generator step events File usr local lib python3 8 site packages dagster databricks databricks pyspark step launcher py line 180 in launch step yield from self step events iterator step context step key databricks run id File usr local lib python3 8 site packages dagster databricks databricks pyspark step launcher py line 222 in step events iterator done = poll run state File usr local lib python3 8 site packages dagster databricks databricks py line 271 in poll run state raise DatabricksError error message ```

Huib Keemink · Answer

^ this is an example of a failed job from today so I really have to go to the databricks logs to figure out what went wrong

Huib Keemink · Answer

For the happy paths stuff works as it should it does work rather nicely but I find I m not too interested in that wink

Huib Keemink · Answer

but it could be that I m just missing a configuration option and this is supported somehow