Make OSS Contribution on M1 easier

Was this while building/installing the grpcio package? That’s by far the most common blocker for m1 development (or just using dagster locally on an M1) and we’re actively working on addressing that now - dockerizing is one possible solution

No there were issues with installing homebrew on the rosetta terminal

Oh interesting - that one was not on my radar, it’s possible that grpcio is the reason we have to point people at rosetta in the first place though

Yeah dagster is fine. It is just the dev env setup that decided to bite me

There’s a meaningful subset of m1 users who can’t even install dagster due to arch issues- that’s the main focus, but the improvements for that may help the dev environment as well

Ah that’s even more of a pain. Completely understand why that’d be a priority for you guys

I lost so much time on this, it’s embarrassing. Python packaging is… getting better, but still very rough, especially when dealing with native libraries. For

grpcio

I was finally able to get it to work on native m1 thanks to this thread using

poetry

and this entry under

[tool.poetry.dependencies]

in

pyproject.toml

grpcio = [
    { version = "^1.51.1", markers = "sys_platform != 'darwin' and platform_machine!='arm64'", source = "pypi" },
    { markers = "sys_platform == 'darwin' and platform_machine=='arm64'", path = 'vendored/grpcio-1.51.1-cp311-cp311-macosx_11_0_arm64.whl' },
]

So, basically, you download that pre-built wheel locally, and point

poetry

at it when it’s on

darwin-aarch64

otherwise, have it use the PyPi dependency. Your mileage may vary with other dependency management tools. For Docker, I wasn’t able to get a working version on alpine due to its use of

musl

so stick to

glibc

-based base images if you want to avoid another world of pain.

We're looking into ways to include our own pre-built wheel of grpcio in dagster until grpcio gets their act together and includes one for M1 macs in the regular installation

Yeah, that would have avoided most of my problems 👍

my understanding is that one is included in the next grpcio release - but a separate upstream grpcio issue is forcing us to include a pin to an earlier grpcio version 😕 So we may still end up needing to include one of our own for a while

I feel your pain! thanks for all your efforts towards making people’s lives easier 🙏

@Todd de Quincey I asked around and I think the main reason we recommend rosetta in that doc was the grpcio issues, we'll probably just take that whole section out of the docs this week since we have a wheel going out for that issue. We have plenty of engineers here internally who develop on dagster without using rosetta

Ah nice!

I just added some experts on the fivetran integration to the PR

Legend 🙇