Hey guys, I’m wondering if some of the pins for dagster-aws and dagster-pyspark may be broken? Seems like they’re being released on pypi under 0.16.x instead of 1.0.x, and the only 1.x release is the yanked 1.0.5 release. Furthermore, I’m new to dagster, but there doesn’t seem to be a 0.16.x series in pypi. If this isn’t the case, is there any guidance to resolve this? I’m wondering if it relates to this older issue, specifically the contents of the versions.py for these packages TLDR: Can’t install dagster-pyspark and dagster-aws via poetry on dagster 1.0.7, may be a bug, but if I’m missing something I’d love a pointer 🙂 Update: I believe I was able to fix via forcing the 0.16.7 install, but haven’t had the change to test extensively.

Hi Zach - the dagster core library is currently at 1.0.x but the integration libraries are at 0.16.x: https://docs.dagster.io/getting-started/releases#dagster-integration-libraries - there should be pins in place so that if you install dagster 1.0.7 and just include 'dagster-aws' without a pin, the right version is installed

That seems to error out while using poetry. There’s a simple workaround by instead running for example

poetry install dagster-aws@~0.16.8

, but sounds like this isn’t expected behavior. Should I create a bug report in the github? Also, sorry for the messy console output 😅

~/broken_poetry_dagster is :package: v0.1.0 via :snake: v3.10.6 on :cloud:  (us-east-1) 
❯ poetry add dagster
Creating virtualenv broken-poetry-dagster-kh3VkVZd-py3.10 in /Users/zpaden/Library/Caches/pypoetry/virtualenvs
Using version ^1.0.8 for dagster

Updating dependencies
Resolving dependencies... (7.4s)

Writing lock file

Package operations: 35 installs, 0 updates, 0 removals

  • Installing markupsafe (2.1.1)
  • Installing six (1.16.0)
  • Installing certifi (2022.6.15)
  • Installing charset-normalizer (2.1.1)
  • Installing grpcio (1.48.0)
  • Installing humanfriendly (10.0)
  • Installing idna (3.3)
  • Installing mako (1.2.2)
  • Installing protobuf (3.20.1)
  • Installing pyparsing (3.0.9)
  • Installing python-dateutil (2.8.2)
  • Installing python-editor (1.0.4)
  • Installing pytzdata (2020.1)
  • Installing sqlalchemy (1.4.41)
  • Installing urllib3 (1.26.12)
  • Installing alembic (1.6.5)
  • Installing click (8.1.3)
  • Installing coloredlogs (14.0)
  • Installing croniter (1.3.7)
  • Installing docstring-parser (0.15)
  • Installing grpcio-health-checking (1.43.0)
  • Installing jinja2 (3.1.2)
  • Installing packaging (21.3)
  • Installing pendulum (2.1.2)
  • Installing pytz (2022.2.1)
  • Installing pyyaml (6.0)
  • Installing requests (2.28.1)
  • Installing rx (1.6.1)
  • Installing tabulate (0.8.10)
  • Installing toposort (1.7)
  • Installing tqdm (4.64.1)
  • Installing typing-compat (0.1.0)
  • Installing typing-extensions (4.3.0)
  • Installing watchdog (2.1.9)
  • Installing dagster (1.0.8)

~/broken_poetry_dagster is :package: v0.1.0 via :snake: v3.10.6 on :cloud:  (us-east-1) took 1m33s 
❯ poetry add dagster-aws
Using version ^1.0.5 for dagster-aws

Updating dependencies
Resolving dependencies... (0.0s)

  SolverProblemError

  Because no versions of dagster-aws match >1.0.5,<2.0.0
   and dagster-aws (1.0.5) depends on dagster (1.0.5), dagster-aws (>=1.0.5,<2.0.0) requires dagster (1.0.5).
  So, because broken-poetry-dagster depends on both dagster (^1.0.8) and dagster-aws (^1.0.5), version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

Could you share the exact combo of versions that it's using?

In the above example, I didn’t specify any. Just ran the following commands:

mkdir broken_dagster
cd broken_dagster
poetry init (Didn't add any deps)
poetry add dagster 
poetry add dagster-aws

dagster resolves to 1.0.8, where as dagster-aws resolved to 1.0.5.

that's my most likely guess, very unfortunate - let me see if i can reproduce

Ahhh seems like I may be using an old version of poetry that doesnt support yanked releases properly. See here

aha

Thanks for the help daniel! I appreciate it 🙂