Hi again wave Meeting some issues on the branch deployment C dagster #dagster-plus

Hi again :wave:, Meeting some issues on the branc...

Charles Couble

02/14/2023, 3:52 PM

Hi again 👋, Meeting some issues on the branch-deployment CD github action. Basically, i have apython dependency looking for a github repository on my setup.py,

Copy code

install_requires=[
        "dagster",
        "dagster-cloud",
        "boto3",
        "pandas",
        'internals @ git+<https://my-private-repo>@github.com/repo.git']

Which works in local, by executing the

pip install -e '.[dev]'

command. Therefore, it seems like the deploy fails on the CI/CD github action with the error shown on the screenshot. (It looks like github actions try to fetch for a local directory) Does anyone have an idea about how to solve this issue ?

Shalabh Chaturvedi

02/14/2023, 5:23 PM

Hi Charles, thanks for the clear bug report regarding treating the remote path as local. We have a fix for that in our release pipeline and and will be rolling it out today. However the other issue here is that this is a private repository and you will need some credentials to check that out in a github workflow. Do you have a username and token in your git+https url to this repo above or do you have some other credentials set up for in the workflow for your private repo?

Shalabh Chaturvedi

02/14/2023, 5:38 PM

Here is a solution that can help with installing the internal repo, even without the local filepath bugfix: 1. In your github workflow, add the following to the top of the steps to checkout and build your private repo.

Copy code

- name: Checkout internal repo
        uses: actions/checkout@v3
        with:
          token: ${{ secrets.GH_PAT }}
          repository: private-org/internal-repo
          path: deps/internal-repo
      
      - name: Build internal repo into a wheel
        run: >
          cd deps/internal-repo &&
          python setup.py bdist_wheel &&
          cp dist/*whl $GITHUB_WORKSPACE/deps
                    
      - name: Configure dependency resolution to use locally built deps/
        run: >
          echo "[global]" > $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "find-links = " >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "    file://$GITHUB_WORKSPACE/deps/" >> $GITHUB_WORKSPACE/deps/pip.conf &&
          echo "PIP_CONFIG_FILE=$GITHUB_WORKSPACE/deps/pip.conf" > $GITHUB_ENV
          && cat $GITHUB_WORKSPACE/deps/pip.conf
          && ls $GITHUB_WORKSPACE/deps/

2. Create a github personal access token and set it as the

GH_PAT

secret for your actions. 3. You may need to fix the "Build internal repo" to cd into the right directory within your internal repo 4. Replace

"internals @ git+https..."

in your setup.py with just

"internals"

. The PIP_CONFIG_FILE points the dependency resolution to use the locally build wheel for the "internals" package. Let us know if this helps.

Charles Couble

02/15/2023, 9:15 AM

Hi Shalbabh, Thanks very much for your answers, I'm currently using a Personnal Access Token (PAT) from Github to authenticate, with no other processes. I will try the solution above and keep you in touch

Shalabh Chaturvedi

02/15/2023, 6:08 PM

Sounds good. I have also added this suggestion here: https://github.com/dagster-io/dagster/discussions/12340#discussioncomment-4976293

Charles Couble

02/16/2023, 3:34 PM

Hey ! Thanks very much for your help 🙏, I had to work around some dependency hell issues, But you fix works like a charm ! No more error regarding my private repository🙂 ! I could not pass the CI, because of the DBT submodule integration, (i've never setup submodules , especially on github actions before), But i will try to dig by myself ! Have a great day

5 Views

Open in Slack

Previous Next