19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Hi all,
Firstly just want to say that Dagster look...
# announcementsp
Peter B
02/17/2021, 11:30 PMHi all,
Firstly just want to say that Dagster looks like an excellent product, so congratulations if any of the developers are on here. I am looking to set it up for my organisation.
We are on Heroku. When I use the Heroku Python build-pack the project doesn't build, as 'the app is not compatible with the buildpack'.
Any idea how I would get the Dagit UI running on there?
I saw the Makefile and Dockerfile, but i'm not clear what I should be looking at.
Would really appreciate any guidance, if it's not too much trouble.
Basically a permanent instance of Dagit so that we can sequence our data pipelines.
I was also tossing up a container but we are quite small and then I'm concerned about losing the state if we pull it down, overkill in terms of setting up the extra storage, etc.?
Thanks,
Pete
n
Noah K
02/18/2021, 12:06 AMDo you get a specific error?
Noah K
02/18/2021, 12:06 AMAlso what does your requirements.txt look like
p
Peter B
02/19/2021, 4:52 AMthanks Noah. It was that I hadn't renamed docs-requirements.txt to requirements.txt
Out of interest, why would it be this separate name in the git repository? This might be a standard convention I'm not aware of.
Peter B
02/19/2021, 4:53 AMUnfortunately the build still failed.
Peter B
02/19/2021, 4:53 AMn
Noah K
02/19/2021, 4:56 AMWhat's in your requirements.txt?
p
Peter B
02/19/2021, 4:57 AMeverything from the docs-requirements.txt from the dagster git repo
Peter B
02/19/2021, 4:57 AMI think though, it may have been that the slug size was 527mb, Heroku max is 500 meg
n
Noah K
02/19/2021, 4:58 AMNot sure what you mean
Noah K
02/19/2021, 4:58 AMAre you trying to just push the dagster repo to heroku?
Noah K
02/19/2021, 4:58 AMThat very will not work
p
Peter B
02/19/2021, 4:58 AMyes
Peter B
02/19/2021, 4:58 AMoh ok
Peter B
02/19/2021, 4:59 AMi basically just want Dagit
n
Noah K
02/19/2021, 4:59 AMDagster is a huge monorepo covering dozens of projects
p
Peter B
02/19/2021, 4:59 AMright
Peter B
02/19/2021, 4:59 AMsorry
Peter B
02/19/2021, 4:59 AMI just want a permanent dagit instance
n
Noah K
02/19/2021, 4:59 AMyou need to make your own stub repo with a requirements.txt that installs dagster, dagit, probably dagster_postgres
Noah K
02/19/2021, 4:59 AMand a procfile that runs it
Noah K
02/19/2021, 4:59 AMI have a similar thing for kubernetes but that won't help you for heroku 🙂
p
Peter B
02/19/2021, 4:59 AMoh ok thanks, genius and legend, thank you!
n
Noah K
02/19/2021, 5:01 AMJust about the only thing I could think of being a problem is that heroku postgres exposes the database config as a URL and I don't think dagster_postgres can parse that?
Noah K
02/19/2021, 5:02 AMIt wants a split out config
p
Peter B
02/19/2021, 5:02 AMoh right
n
Noah K
02/19/2021, 5:02 AMNoah K
02/19/2021, 5:02 AMSo you might need an entrypoint script to parse the URL to separate env vars and then exec dagit
p
Peter B
02/19/2021, 5:04 AMyes ok that makes sense, thanks!!
Peter B
02/19/2021, 5:04 AMappreciate your help
Peter B
02/19/2021, 5:09 AMwould I still install the libraries that I need, e.g. dagster-dbt?
Peter B
02/19/2021, 5:09 AMlike keep in requirements.txt, or was there an issue specifically that I was pushing the libraries to heroku in that context
n
Noah K
02/19/2021, 5:22 AMDepends on how you're planning to launch runs?
Noah K
02/19/2021, 5:22 AMWhere will your jobs be executing
p
Peter B
02/19/2021, 5:27 AMlooking to launch everything from local command line
Peter B
02/19/2021, 5:27 AMbut execute and run everything off the heroku app
Peter B
02/19/2021, 5:27 AMand have access to the dagit UI via web access (obviously with the port security access etc. configured)
n
Noah K
02/19/2021, 5:31 AMEither your jobs must be very tiny or dynos have gotten a lot bigger
Noah K
02/19/2021, 5:31 AMIIRC the default dyno size was like 1GB of RAM?
p
Peter B
02/22/2021, 12:16 AMHmm there is a 14 gig but good point.
n
Noah K
02/22/2021, 12:23 AMIf you want to run stuff in the dyno then yes, it all has to be installed there
Noah K
02/22/2021, 12:23 AMThis doesn't seem like a good use of Heroku though
10 Views