https://dagster.io/ logo
#ask-ai
Title
# ask-ai
s

Selim Raboudi

02/28/2024, 9:44 PM
In Dagster, can I add as a dependency a python package that is hosted on a private github repo ?
s

Scout

02/28/2024, 9:44 PM
Yes, you can add a Python package that is hosted on a private GitHub repository as a dependency in your Dagster project. The documentation provides a method for including a private package as a dependency in a Dagster Cloud project, particularly when using fast deploys in your workflow. Here's a summary of the steps you would need to follow: 1. In your GitHub Actions workflow file (typically
.github/workflows/deploy.yml
), add steps to check out the private repository and build a wheel from it. You would use the
actions/checkout
GitHub Action with a personal access token to access the private repository. 2. Build a wheel for the private package and place it in a directory that can be referenced later in the workflow. 3. Configure dependency resolution to use the wheel you built by creating a
pip.conf
file in the workflow that points to the directory containing the wheel. 4. Create a GitHub personal access token with the appropriate permissions and set it as a secret in your GitHub repository settings. This token will be used by the GitHub Actions workflow to access the private repository. 5. In your Dagster project's
setup.py
, add the private package to the
install_requires
section, ensuring that the package name matches the name of your private Python package. Here is an example of how you might configure your GitHub Actions workflow to include a private package:
Copy code
yaml - name: Checkout internal repository   uses: actions/checkout@v3   with:     token: ${{ secrets.GH_PAT }}     repository: my-org/private-repo     path: deps/private-repo     ref: some-branch   # optional to check out a specific branch  - name: Build a wheel   run: |     cd deps/private-repo &&     python setup.py bdist_wheel &&     mkdir -p $GITHUB_WORKSPACE/deps &&     cp dist/*whl $GITHUB_WORKSPACE/deps  - name: Configure dependency resolution to use the wheel built above   run: |     echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&     echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&     echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&     echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV
And in your
setup.py
, you would add something like this:
Copy code
python setup(     ...     install_requires=[         "dagster",         "dagster-cloud",         "private-package",   # add this line - must match your private Python package name     ],     ... )
Please note that this method is specific to workflows that use GitHub Actions and Dagster Cloud. If you are not using Dagster Cloud or GitHub Actions, you may need to adapt these steps to fit your particular deployment process.
4 Views