Hi Team We have a common packages (or first party dependencies) which consists of the common functionalities. We have multiple dagster python projects (let's call as user code package), these uses the methods/functions of common packages. If we our user code package is using dagster_databricks_step_launcher it only zips the user code package. It does not include common packages (or first party dependencies) as a result the jobs running in databricks throw ModuleNotFound error. How to handle this scenario ?

Hi Tushar, This sounds like a Python environment configuration question rather than anything dagster-specific. Are you building docker images for your user coed packages? You need to make sure to install all the necessary dependencies into the python environment for each package. If you list them in the package

setup.py

or

requirements.txt

, that should solve it, but it’s hard to help without knowing anything else about how you are deploying.

Yes we are building docker images with all the third party dependencies (publicly available packages), first party dependencies (common package) and user code package installed. When we want to use dagster_databricks_step_launcher for running databricks jobs, it only zips the user code package and make it available for Databricks cluster. (I am not sure how databricks step launcher uploads this zip file to DBFS) We are not worried about the third party packages because we can mention the packages and version in run_config. But the first party packages are not available in PyPI. So the question how can we configure dagster to zip the first party dependencies as well ?

OK I think I see the problem-- I’ve put in a request for an engineer with more familiarity with our databricks integration to respond here.

Apologies for chasing @sean, but I would like to know Dagster team's view on this.