is it possible to use a PEX binary as an entrypoint in a user code image for a hybrid cloud deployment? it feels like the answer would be no, seeing as I think the entrypoint gets overwritten by Dagster when the container is called correct (we're deployed on ECS if that matters). but maybe I'm missing something?

Hey Zach - there's an "executable_path" config key in cloud that you can set in your dagster_cloud.yaml that will determine which executable it uses - as long as its a Python executable that Dagster can append "-m dagster api execute-run" (which I believe would work with a PEX) then that should work

absolutely righteous, forgot about that option I haven't updated a user code location in a while. PEX seems really really cool, I'll be exploring this more soon! thanks!

double-checking, this is a PEX that you're building, right? we build them in serverless but i wouldn't recommend trying to use one of those in Hybrid yet

ah okay. yeah this is a PEX I'm building, I'm looking to integrate Pantsbuild to streamline our monorepo build process and it has really tight support for generating PEX files as build artifacts. but if it's not really appropriate to do use PEX in Hybrid yet then I can still build my images just with normal wheel installs into the docker image

PEX in hybrid is fine as long as you're building it and including it in the image yourself

okay cool, yeah that would be the plan

Is adding pex builds, quick builds etc on the roadmap for hybrid?

It's on the medium-term roadmap, but it's not actively being worked on currently

@Zach The other option is using Pants/PEX inside docker. It can give you quite a small docker image size. For example I currently build these two lambda image using • AWS provided base docker image in multi stage • Another lambda image just PEX with base python image in multi stage AWS lambda image is like 1.2 GB and PEX one is like 300 MB

@Shantanu are you suggesting running the pants packaging commands inside the docker image? wouldn't you have to copy the whole project into the docker image first?

If your host and docker env is not very distinct, you can build it locally and push it to your docker registry. Issues arises when you work on Mac and build a docker image which of course breaks things because PEX Is platform dependent. And no you don’t have to copy all of your code base inside the docker image. I am not sure if you are already aware of this article https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/

I see, I think we're talking about the same approach then. I haven't seen that article, thanks for the link though!

Issues arises when you work on Mac and build a docker image which of course breaks things because PEX Is platform dependent.

You can cross build pex files as along as wheels are available for all dependencies. The flag to use is

--platform

or

--complete-platform

. These are also available in the

pex_binary

rule: https://www.pantsbuild.org/docs/reference-pex_binary#codecomplete_platformscode Looks like pants also has new support for cross builds using a docker builder environment: https://www.pantsbuild.org/docs/environments#use-a-docker_environment-to-build-the-inputs-to-a-docker_image