https://dagster.io/ logo
Join Slack
Powered by
Hi, I have a question about python dependencies a...
# ask-community
s
Hi, I have a question about python dependencies and decoupling our pipelines from `dagster`'s own requirements. Looking at the docs on multi-repo + multi-env, I understand that we can run each pipeline in their own venv. However, in order for us to create pipelines, we have to do 
pip install dagster
which will bring along dagster's dependencies. Then, aren't we implicitly tied to dagster's dependencies anyways? Thanks! cc @Serj Bilokhatniuk
d
Hi Stanley - it's true that you'll need to bring in dagster's dependencies. Is there a particular one there that causes problems for you? The benefit of the multi-repo stuff is less about avoiding dagster's dependencies (which we try to keep pretty light) and more about letting you run different sets of code in different environments that might have different conflicting dependencies.
s
alembic and sqlalchemy are non-trial dependencies for any project to have locked
pywin32 is a nice thing to carry around too = )
d
are you on windows?
(assuming you saw that pywin32 is only set if the platform is windows)
s
no, and that's my point = )
d
right - if you're on a non-windows platform, that dependency won't be included
s
I am just poking holes, there, but sqlalchemy is realy big deal, while other deps are less likely, but not impossible to cause dep conflicts
would dagster project consider vendoring its deps?
d
When you say locked - this is great feedback, just trying to nail down the details. If the libraries were there but unpinned, would that still be a problem? Or is the problem that they are pinned to specific versions, and broadening the constraints would eliminate the problem?
s
it's the pinned versions, of course, but also I don't think it's realistic for dagster not to pin the deps, like it would be unrealistic for any long-term project, even it it's just 
x>=1
selector. It's even worse for dagit, with flask and a kitchen sink in there
most of the widely used Python project vendor their deps, or used to vendor, until deps became part of std lib or were obsoleted
not that this is not true
Dagster’s repository model lets you isolate codebases so that problems in one job don’t bring down the rest. Each job can have its own package dependencies and Python version. Jobs are run in isolated processes so user code issues can't bring the system down.
from: https://github.com/dagster-io/dagster#avoid-dependency-nightmares
but there is a BIG caveat that repos still may need to use same (or close enough) dagster version if they share the control plane and also be in harmony with dagster's own requirements
d
Ah so for the first thing - they don't actually have to have the same version as the control plane. They communicate over an API that we test for backwards compatbility
So you can have your code on an older version and just upgrade the control plane, and vice versa
s
I think only anecdotally, but we saw 0.12 repo not working wiht 0.14 control plane, but we might have fudged something up
d
Got it - we'd be happy to look more closely at that if you have an error or more details about how it didn't work
s
I think we just decided not to bank on backward compatibility
using dagster has a distinct feel of being a beta tester and reminds me a lot of the earlier Linux years, but then I can't complain for the money we pay for it either (which is nothing)
and I know it's not trivial to do anything about the underlying architecture of dagster to reshuffle dependencies, and if anything can be done realistically I think its just vendoring or nothing
d
Thanks for the feedback - I don't think there are any plans to add vendoring for now, but I will pass the feedback along to the rest of the team
2 Views
Open in Slack
PreviousNext
Powered by