https://dagster.io/ logo
#ask-community
Title
# ask-community
l

Luuk Derksen

05/10/2023, 2:26 PM
Hi 👋 I am trying to install a private package (using a private github-repo for that) inside the Dagster deploy. I was following this thread and this discussion thusfar and it now breaks on the final github action step. The suggested code added:
Copy code
# For docs on installing private packages: 
      # <https://github.com/dagster-io/dagster/discussions/12340>
      - name: Checkout orbiit_common
        uses: actions/checkout@v3
        with:
          token: ${{ secrets.GHPR_TOKEN }}
          repository: orbiitai/data-python-libraries
          path: orbiit_common
          ref: main

      - name: Build the wheel for orbiit_common
        run: >
          cd orbiit_common/orbiit_common &&
          python setup.py bdist_wheel &&
          mkdir $GITHUB_WORKSPACE/deps &&
          cp dist/*whl $GITHUB_WORKSPACE/deps &&
          touch $GITHUB_WORKSPACE/deps/pip.conf

      - name: Configure dependency resolution to use the orbiit_common wheel built above
        run: >
          echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV
Copy code
- name: Build and deploy Python executable
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: dagster-io/dagster-cloud-action/actions/build_deploy_python_executable@pex-v0.1
        with:
          dagster_cloud_file: "$GITHUB_WORKSPACE/project-repo/dagster_cloud.yaml"
          build_output_dir: "$GITHUB_WORKSPACE/build"
          python_version: "3.10"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Fails with. It seems that the
pip.conf
is not used inside the action?
Copy code
STDERR:\nERROR: Could not find a version that satisfies the requirement 
orbiit_common (from versions: none)\nERROR: No matching distribution found for 
orbiit_common\n\n")
Any suggestions/advice?
s

Shalabh Chaturvedi

05/10/2023, 4:46 PM
Hi Luuk, can you add the following line to ensure that the wheel is being built locally and the pip.conf looks correct:
Copy code
name: Configure dependency resolution to use the orbiit_common wheel built above
        run: >
          echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV
######### ADD THIS:
          && ls $GITHUB_WORKSPACE/deps && cat $GITHUB_WORKSPACE/deps/pip.conf
Also can you confirm you have
ENABLE_FAST_DEPLOYS
set to
true
in the top level env?
l

Luuk Derksen

05/16/2023, 3:47 PM
@Shalabh Chaturvedi I added it and ran:
Copy code
- name: Configure dependency resolution to use the orbiit_common wheel built above
        run: >
          echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV && 
          ls $GITHUB_WORKSPACE/deps && cat $GITHUB_WORKSPACE/deps/pip.conf
Resulting in
Copy code
orbiit_common-0.6.2-py3-none-any.whl
pip.conf
[global]
find-links = 
    file:///home/runner/work/data-dagster/data-dagster/deps/
The failure happens in the following step when it pulls the container and tries to build/install the dependencies where it tries to grab the requirement. This repo has NO published packages, we just want to build it from the repo itself. Is this something that should be changed in the
setup.py
file to point to the right path?
Copy code
--requirement                                   │ │
│ │                          /output/deps-requirements-ed5c4484a036786aa905… │ │
│ │                          --index-url <https://pypi.org/simple> --retries 5 │ │
│ │                          --timeout 15 exited with 1 and STDERR:\nERROR:  │ │
│ │                          Could not find a version that satisfies the     │ │
│ │                          requirement orbiit_common (from versions:       │ │
│ │                          none)\nERROR: No matching distribution found    │ │
│ │                          for orbiit_common\n\n")
s

Shalabh Chaturvedi

05/16/2023, 4:25 PM
We build and deploy our internal unreleased package versions using the
pip.conf
method above. I'll take a closer look to see if there are any subtle differences. Can you please share the
install_requires
section of your
setup.py
? I understand the orbiit_common is an unreleased package I expect this to still work since the wheel is available locally.
So I think having no published packages at all might be the issue here. When we use pip.conf we have at least some old version on PyPI. You can use relative paths in a requirements.txt instead. For example, the following should work:
Copy code
project-repo
  dagster_cloud.yaml
  setup.py
  requirements.txt # add this file with `../orbiit_common
orbiit_common # check this out as above
  setup.py
You can also use direct references in setup.py, eg
"orbiit_commont @ file:///.../orbiit_common.whl
, however I dont believe this accepts relative paths.
l

Luuk Derksen

05/17/2023, 5:42 AM
Neither works at this moment @Shalabh Chaturvedi but I am wondering now if it has to do with the order of the github actions steps. I put the wheel building at the top, should this happen between the project checkout and building of the executable?
Copy code
steps:

      # For docs on installing private packages: 
      # <https://github.com/dagster-io/dagster/discussions/12340>
      - name: Checkout orbiit_common
        uses: actions/checkout@v3
        with:
          token: ${{ secrets.GHPR_TOKEN }}
          repository: orbiitai/data-python-libraries
          path: orbiit_common
          ref: main

      - name: Build the wheel for orbiit_common
        run: >
          cd orbiit_common/orbiit_common &&
          python setup.py bdist_wheel &&
          mkdir $GITHUB_WORKSPACE/deps &&
          cp dist/*whl $GITHUB_WORKSPACE/deps &&
          touch $GITHUB_WORKSPACE/deps/pip.conf

      - name: Configure dependency resolution to use the orbiit_common wheel built above
        run: >
          echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV && 
          ls $GITHUB_WORKSPACE/deps && cat $GITHUB_WORKSPACE/deps/pip.conf

      - name: Parse cloud workspace
        if: env.ENABLE_FAST_DEPLOYS != 'true'
        id: parse-workspace
        uses: dagster-io/dagster-cloud-action/actions/utils/parse_workspace@v0.1
        with:
          dagster_cloud_file: dagster_cloud.yaml

      - name: Checkout
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: actions/checkout@v3
        with:
          ref: ${{ github.head_ref }}
          path: project-repo

      - name: Build and deploy Python executable
        if: env.ENABLE_FAST_DEPLOYS == 'true'
        uses: dagster-io/dagster-cloud-action/actions/build_deploy_python_executable@pex-v0.1
        with:
          dagster_cloud_file: "$GITHUB_WORKSPACE/project-repo/dagster_cloud.yaml"
          build_output_dir: "$GITHUB_WORKSPACE/build"
          python_version: "3.10"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
The most telling error is probably this. When adding
./data-dagster/data-dagster/deps/orbiit_common-0.6.2-py3-none-any.whl
to the
requirements.txt
it throws this error
Copy code
ValueError: Could not find local directory 
'/home/runner/work/data-dagster/data-dagster/project-repo/data-dagster/data-dags
ter/deps/orbiit_common-0.6.2-py3-none-any.whl' referenced in requirement 
'./data-dagster/data-dagster/deps/orbiit_common-0.6.2-py3-none-any.whl'
Now I added
./deps/orbiit_common-0.6.2-py3-none-any.whl
to the
requirements.txt
and it throws
Copy code
ValueError: Could not find local directory 
'/home/runner/work/data-dagster/data-dagster/project-repo/deps/orbiit_common-0.6
.2-py3-none-any.whl' referenced in requirement 
'./deps/orbiit_common-0.6.2-py3-none-any.whl'
From the
ls
in earlier steps the file should reside in
file:///home/runner/work/data-dagster/data-dagster/deps/orbiit_common-0.6.2-py3-none-any.whl
s

Shalabh Chaturvedi

05/17/2023, 6:01 PM
Can you change that to two leading dots?
../deps/orbiit_common-0.6.2-py3-none-any.whl
. If you check out the shared code locally, you don't need to build the wheel. Give the above paths you can just add
../orbiit_common/orbiit_common
to your requirements.txt. Your directory structure would look like this:
Copy code
$GITHUB_WORKSPACE
    orbiit_common/
        orbiit_common/
            setup.py
            ...

    project-repo/
        requirements.txt # contains '../orbiit_common/orbiit_common'
        ...
This should be able to find the orbiit_common project and build it.
👀 1
12 Views