https://dagster.io/ logo
#ask-community
Title
# ask-community
z

Zach

03/23/2023, 4:54 PM
is it possible to use a PEX binary as an entrypoint in a user code image for a hybrid cloud deployment? it feels like the answer would be no, seeing as I think the entrypoint gets overwritten by Dagster when the container is called correct (we're deployed on ECS if that matters). but maybe I'm missing something?
d

daniel

03/23/2023, 4:58 PM
Hey Zach - there's an "executable_path" config key in cloud that you can set in your dagster_cloud.yaml that will determine which executable it uses - as long as its a Python executable that Dagster can append "-m dagster api execute-run" (which I believe would work with a PEX) then that should work
z

Zach

03/23/2023, 4:59 PM
absolutely righteous, forgot about that option I haven't updated a user code location in a while. PEX seems really really cool, I'll be exploring this more soon! thanks!
d

daniel

03/23/2023, 5:05 PM
double-checking, this is a PEX that you're building, right? we build them in serverless but i wouldn't recommend trying to use one of those in Hybrid yet
z

Zach

03/23/2023, 5:07 PM
ah okay. yeah this is a PEX I'm building, I'm looking to integrate Pantsbuild to streamline our monorepo build process and it has really tight support for generating PEX files as build artifacts. but if it's not really appropriate to do use PEX in Hybrid yet then I can still build my images just with normal wheel installs into the docker image
d

daniel

03/23/2023, 5:16 PM
PEX in hybrid is fine as long as you're building it and including it in the image yourself
z

Zach

03/23/2023, 5:17 PM
okay cool, yeah that would be the plan
m

Matt Clarke

03/23/2023, 5:21 PM
Is adding pex builds, quick builds etc on the roadmap for hybrid?
d

daniel

03/23/2023, 5:23 PM
It's on the medium-term roadmap, but it's not actively being worked on currently
👍 1
s

Shantanu

03/23/2023, 5:38 PM
@Zach The other option is using Pants/PEX inside docker. It can give you quite a small docker image size. For example I currently build these two lambda image using • AWS provided base docker image in multi stage • Another lambda image just PEX with base python image in multi stage AWS lambda image is like 1.2 GB and PEX one is like 300 MB
z

Zach

03/23/2023, 5:43 PM
@Shantanu are you suggesting running the pants packaging commands inside the docker image? wouldn't you have to copy the whole project into the docker image first?
s

Shantanu

03/23/2023, 5:49 PM
If your host and docker env is not very distinct, you can build it locally and push it to your docker registry. Issues arises when you work on Mac and build a docker image which of course breaks things because PEX Is platform dependent. And no you don’t have to copy all of your code base inside the docker image. I am not sure if you are already aware of this article https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/
z

Zach

03/23/2023, 5:51 PM
I see, I think we're talking about the same approach then. I haven't seen that article, thanks for the link though!
s

Shalabh Chaturvedi

03/23/2023, 8:43 PM
Issues arises when you work on Mac and build a docker image which of course breaks things because PEX Is platform dependent.
You can cross build pex files as along as wheels are available for all dependencies. The flag to use is
--platform
or
--complete-platform
. These are also available in the
pex_binary
rule: https://www.pantsbuild.org/docs/reference-pex_binary#codecomplete_platformscode Looks like pants also has new support for cross builds using a docker builder environment: https://www.pantsbuild.org/docs/environments#use-a-docker_environment-to-build-the-inputs-to-a-docker_image
4 Views