https://dagster.io/ logo
p

Peter B

02/17/2021, 11:30 PM
Hi all, Firstly just want to say that Dagster looks like an excellent product, so congratulations if any of the developers are on here. I am looking to set it up for my organisation. We are on Heroku. When I use the Heroku Python build-pack the project doesn't build, as 'the app is not compatible with the buildpack'. Any idea how I would get the Dagit UI running on there? I saw the Makefile and Dockerfile, but i'm not clear what I should be looking at. Would really appreciate any guidance, if it's not too much trouble. Basically a permanent instance of Dagit so that we can sequence our data pipelines. I was also tossing up a container but we are quite small and then I'm concerned about losing the state if we pull it down, overkill in terms of setting up the extra storage, etc.? Thanks, Pete
n

Noah K

02/18/2021, 12:06 AM
Do you get a specific error?
Also what does your requirements.txt look like
p

Peter B

02/19/2021, 4:52 AM
thanks Noah. It was that I hadn't renamed docs-requirements.txt to requirements.txt Out of interest, why would it be this separate name in the git repository? This might be a standard convention I'm not aware of.
Unfortunately the build still failed.
n

Noah K

02/19/2021, 4:56 AM
What's in your requirements.txt?
p

Peter B

02/19/2021, 4:57 AM
everything from the docs-requirements.txt from the dagster git repo
I think though, it may have been that the slug size was 527mb, Heroku max is 500 meg
n

Noah K

02/19/2021, 4:58 AM
Not sure what you mean
Are you trying to just push the dagster repo to heroku?
That very will not work
p

Peter B

02/19/2021, 4:58 AM
yes
oh ok
i basically just want Dagit
n

Noah K

02/19/2021, 4:59 AM
Dagster is a huge monorepo covering dozens of projects
p

Peter B

02/19/2021, 4:59 AM
right
sorry
I just want a permanent dagit instance
n

Noah K

02/19/2021, 4:59 AM
you need to make your own stub repo with a requirements.txt that installs dagster, dagit, probably dagster_postgres
and a procfile that runs it
I have a similar thing for kubernetes but that won't help you for heroku 🙂
p

Peter B

02/19/2021, 4:59 AM
oh ok thanks, genius and legend, thank you!
n

Noah K

02/19/2021, 5:01 AM
Just about the only thing I could think of being a problem is that heroku postgres exposes the database config as a URL and I don't think dagster_postgres can parse that?
It wants a split out config
p

Peter B

02/19/2021, 5:02 AM
oh right
So you might need an entrypoint script to parse the URL to separate env vars and then exec dagit
p

Peter B

02/19/2021, 5:04 AM
yes ok that makes sense, thanks!!
appreciate your help
would I still install the libraries that I need, e.g. dagster-dbt?
like keep in requirements.txt, or was there an issue specifically that I was pushing the libraries to heroku in that context
n

Noah K

02/19/2021, 5:22 AM
Depends on how you're planning to launch runs?
Where will your jobs be executing
p

Peter B

02/19/2021, 5:27 AM
looking to launch everything from local command line
but execute and run everything off the heroku app
and have access to the dagit UI via web access (obviously with the port security access etc. configured)
n

Noah K

02/19/2021, 5:31 AM
Either your jobs must be very tiny or dynos have gotten a lot bigger
IIRC the default dyno size was like 1GB of RAM?
p

Peter B

02/22/2021, 12:16 AM
Hmm there is a 14 gig but good point.
n

Noah K

02/22/2021, 12:23 AM
If you want to run stuff in the dyno then yes, it all has to be installed there
This doesn't seem like a good use of Heroku though
9 Views