https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
t

Todd de Quincey

01/14/2023, 3:32 PM
Make OSS Contribution on M1 easier
👍 1
Hi, I am using the
build_fivetran_assets
function and noticed that there isn’t the ability to pass in the
group_name
to the assets. So I created this small (forked) PR which I was planning to submit to Dagster OSS. However, after battling with trying to setup the local environment for over an hour, I’ve given up. Which is quite unfortunate. Is there a technical reason why the dev environment isn’t dockerised? Side note, can the changes that I have on this forked branch be implemented?
P.S. I am happy to submit the PR that I have forked, but as I don’t have the dev env setup, I haven’t run the tests.
d

daniel

01/14/2023, 4:00 PM
Was this while building/installing the grpcio package? That’s by far the most common blocker for m1 development (or just using dagster locally on an M1) and we’re actively working on addressing that now - dockerizing is one possible solution
For now feel free to submit the PR, sorry for the trouble here
t

Todd de Quincey

01/14/2023, 4:01 PM
No there were issues with installing homebrew on the rosetta terminal
I like M1s, but the pain they cause is real
Great to hear that there is something in the works for this 🙂
d

daniel

01/14/2023, 4:03 PM
Oh interesting - that one was not on my radar, it’s possible that grpcio is the reason we have to point people at rosetta in the first place though
Dagster installs ok for you locally on the m1 though?
t

Todd de Quincey

01/14/2023, 4:04 PM
Yeah dagster is fine. It is just the dev env setup that decided to bite me
d

daniel

01/14/2023, 4:07 PM
There’s a meaningful subset of m1 users who can’t even install dagster due to arch issues- that’s the main focus, but the improvements for that may help the dev environment as well
D 1
t

Todd de Quincey

01/14/2023, 4:09 PM
Ah that’s even more of a pain. Completely understand why that’d be a priority for you guys
That’s the PR. As mentioned in the description, the unit test hasn’t been run due to the above. So it might need some work https://github.com/dagster-io/dagster/pull/11718
p

Philippe Laflamme

01/15/2023, 10:20 PM
I lost so much time on this, it’s embarrassing. Python packaging is… getting better, but still very rough, especially when dealing with native libraries. For
grpcio
I was finally able to get it to work on native m1 thanks to this thread using
poetry
and this entry under
[tool.poetry.dependencies]
in
pyproject.toml
Copy code
grpcio = [
    { version = "^1.51.1", markers = "sys_platform != 'darwin' and platform_machine!='arm64'", source = "pypi" },
    { markers = "sys_platform == 'darwin' and platform_machine=='arm64'", path = 'vendored/grpcio-1.51.1-cp311-cp311-macosx_11_0_arm64.whl' },
]
So, basically, you download that pre-built wheel locally, and point
poetry
at it when it’s on
darwin-aarch64
otherwise, have it use the PyPi dependency. Your mileage may vary with other dependency management tools. For Docker, I wasn’t able to get a working version on alpine due to its use of
musl
so stick to
glibc
-based base images if you want to avoid another world of pain.
d

daniel

01/15/2023, 10:21 PM
We're looking into ways to include our own pre-built wheel of grpcio in dagster until grpcio gets their act together and includes one for M1 macs in the regular installation
🎉 1
p

Philippe Laflamme

01/15/2023, 10:22 PM
Yeah, that would have avoided most of my problems 👍
d

daniel

01/15/2023, 10:23 PM
my understanding is that one is included in the next grpcio release - but a separate upstream grpcio issue is forcing us to include a pin to an earlier grpcio version 😕 So we may still end up needing to include one of our own for a while
😭 1
p

Philippe Laflamme

01/15/2023, 10:24 PM
I feel your pain! thanks for all your efforts towards making people’s lives easier 🙏
d

daniel

01/18/2023, 4:43 PM
@Todd de Quincey I asked around and I think the main reason we recommend rosetta in that doc was the grpcio issues, we'll probably just take that whole section out of the docs this week since we have a wheel going out for that issue. We have plenty of engineers here internally who develop on dagster without using rosetta
t

Todd de Quincey

01/18/2023, 7:58 PM
Ah nice!
Thanks for circling back to me on this. Much appreciated
re that PR, is this something that can be looked at? I am running a forked version of Dagster, which I’d like to avoid
d

daniel

01/18/2023, 8:04 PM
I just added some experts on the fivetran integration to the PR
t

Todd de Quincey

01/18/2023, 8:35 PM
Legend 🙇
3 Views