Hi 👋 I am trying to install a private package (using a private github-repo for that) inside the Dagster deploy. I was following this thread and this discussion thusfar and it now breaks on the final github action step. The suggested code added:

# For docs on installing private packages: 
      # <https://github.com/dagster-io/dagster/discussions/12340>
      - name: Checkout orbiit_common
        uses: actions/checkout@v3
        with:
          token: ${{ secrets.GHPR_TOKEN }}
          repository: orbiitai/data-python-libraries
          path: orbiit_common
          ref: main

      - name: Build the wheel for orbiit_common
        run: >
          cd orbiit_common/orbiit_common &&
          python setup.py bdist_wheel &&
          mkdir $GITHUB_WORKSPACE/deps &&
          cp dist/*whl $GITHUB_WORKSPACE/deps &&
          touch $GITHUB_WORKSPACE/deps/pip.conf

      - name: Configure dependency resolution to use the orbiit_common wheel built above
        run: >
          echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV

- name: Build and deploy Python executable
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: dagster-io/dagster-cloud-action/actions/build_deploy_python_executable@pex-v0.1
        with:
          dagster_cloud_file: "$GITHUB_WORKSPACE/project-repo/dagster_cloud.yaml"
          build_output_dir: "$GITHUB_WORKSPACE/build"
          python_version: "3.10"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Fails with. It seems that the

pip.conf

is not used inside the action?

STDERR:\nERROR: Could not find a version that satisfies the requirement 
orbiit_common (from versions: none)\nERROR: No matching distribution found for 
orbiit_common\n\n")

Any suggestions/advice?

Hi Luuk, can you add the following line to ensure that the wheel is being built locally and the pip.conf looks correct:

name: Configure dependency resolution to use the orbiit_common wheel built above
        run: >
          echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV
######### ADD THIS:
          && ls $GITHUB_WORKSPACE/deps && cat $GITHUB_WORKSPACE/deps/pip.conf

Also can you confirm you have

ENABLE_FAST_DEPLOYS

set to

true

in the top level env?

@Shalabh Chaturvedi I added it and ran:

- name: Configure dependency resolution to use the orbiit_common wheel built above
        run: >
          echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV && 
          ls $GITHUB_WORKSPACE/deps && cat $GITHUB_WORKSPACE/deps/pip.conf

Resulting in

orbiit_common-0.6.2-py3-none-any.whl
pip.conf
[global]
find-links = 
    file:///home/runner/work/data-dagster/data-dagster/deps/

The failure happens in the following step when it pulls the container and tries to build/install the dependencies where it tries to grab the requirement. This repo has NO published packages, we just want to build it from the repo itself. Is this something that should be changed in the

setup.py

file to point to the right path?

--requirement                                   │ │
│ │                          /output/deps-requirements-ed5c4484a036786aa905… │ │
│ │                          --index-url <https://pypi.org/simple> --retries 5 │ │
│ │                          --timeout 15 exited with 1 and STDERR:\nERROR:  │ │
│ │                          Could not find a version that satisfies the     │ │
│ │                          requirement orbiit_common (from versions:       │ │
│ │                          none)\nERROR: No matching distribution found    │ │
│ │                          for orbiit_common\n\n")

We build and deploy our internal unreleased package versions using the

pip.conf

method above. I'll take a closer look to see if there are any subtle differences. Can you please share the

install_requires

section of your

setup.py

? I understand the orbiit_common is an unreleased package I expect this to still work since the wheel is available locally.

Neither works at this moment @Shalabh Chaturvedi but I am wondering now if it has to do with the order of the github actions steps. I put the wheel building at the top, should this happen between the project checkout and building of the executable?

steps:

      # For docs on installing private packages: 
      # <https://github.com/dagster-io/dagster/discussions/12340>
      - name: Checkout orbiit_common
        uses: actions/checkout@v3
        with:
          token: ${{ secrets.GHPR_TOKEN }}
          repository: orbiitai/data-python-libraries
          path: orbiit_common
          ref: main

      - name: Build the wheel for orbiit_common
        run: >
          cd orbiit_common/orbiit_common &&
          python setup.py bdist_wheel &&
          mkdir $GITHUB_WORKSPACE/deps &&
          cp dist/*whl $GITHUB_WORKSPACE/deps &&
          touch $GITHUB_WORKSPACE/deps/pip.conf

      - name: Configure dependency resolution to use the orbiit_common wheel built above
        run: >
          echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV && 
          ls $GITHUB_WORKSPACE/deps && cat $GITHUB_WORKSPACE/deps/pip.conf

      - name: Parse cloud workspace
        if: env.ENABLE_FAST_DEPLOYS != 'true'
        id: parse-workspace
        uses: dagster-io/dagster-cloud-action/actions/utils/parse_workspace@v0.1
        with:
          dagster_cloud_file: dagster_cloud.yaml

      - name: Checkout
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}
          path: project-repo

      - name: Build and deploy Python executable
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: dagster-io/dagster-cloud-action/actions/build_deploy_python_executable@pex-v0.1
        with:
          dagster_cloud_file: "$GITHUB_WORKSPACE/project-repo/dagster_cloud.yaml"
          build_output_dir: "$GITHUB_WORKSPACE/build"
          python_version: "3.10"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Can you change that to two leading dots?

../deps/orbiit_common-0.6.2-py3-none-any.whl

. If you check out the shared code locally, you don't need to build the wheel. Give the above paths you can just add

../orbiit_common/orbiit_common

to your requirements.txt. Your directory structure would look like this:

$GITHUB_WORKSPACE
    orbiit_common/
        orbiit_common/
            setup.py
            ...

    project-repo/
        requirements.txt # contains '../orbiit_common/orbiit_common'
        ...

This should be able to find the orbiit_common project and build it.