In Dagster, can I add as a dependency a python package that is hosted on a private github repo ?

Yes, you can add a Python package that is hosted on a private GitHub repository as a dependency in your Dagster project. The documentation provides a method for including a private package as a dependency in a Dagster Cloud project, particularly when using fast deploys in your workflow. Here's a summary of the steps you would need to follow: 1. In your GitHub Actions workflow file (typically

.github/workflows/deploy.yml

), add steps to check out the private repository and build a wheel from it. You would use the

actions/checkout

GitHub Action with a personal access token to access the private repository. 2. Build a wheel for the private package and place it in a directory that can be referenced later in the workflow. 3. Configure dependency resolution to use the wheel you built by creating a

pip.conf

file in the workflow that points to the directory containing the wheel. 4. Create a GitHub personal access token with the appropriate permissions and set it as a secret in your GitHub repository settings. This token will be used by the GitHub Actions workflow to access the private repository. 5. In your Dagster project's

setup.py

, add the private package to the

install_requires

section, ensuring that the package name matches the name of your private Python package. Here is an example of how you might configure your GitHub Actions workflow to include a private package:

yaml - name: Checkout internal repository   uses: actions/checkout@v3   with:     token: ${{ secrets.GH_PAT }}     repository: my-org/private-repo     path: deps/private-repo     ref: some-branch   # optional to check out a specific branch  - name: Build a wheel   run: |     cd deps/private-repo &&     python setup.py bdist_wheel &&     mkdir -p $GITHUB_WORKSPACE/deps &&     cp dist/*whl $GITHUB_WORKSPACE/deps  - name: Configure dependency resolution to use the wheel built above   run: |     echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&     echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&     echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&     echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV

And in your

setup.py

, you would add something like this:

python setup(     ...     install_requires=[         "dagster",         "dagster-cloud",         "private-package",   # add this line - must match your private Python package name     ],     ... )

Please note that this method is specific to workflows that use GitHub Actions and Dagster Cloud. If you are not using Dagster Cloud or GitHub Actions, you may need to adapt these steps to fit your particular deployment process.