19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#dagster-support
Title
# dagster-support
s
Stanley Yang
05/10/2022, 7:19 PMHi,
I have a question about python dependencies and decoupling our pipelines from `dagster`'s own requirements.
Looking at the docs on multi-repo + multi-env, I understand that we can run each pipeline in their own venv.
However, in order for us to create pipelines, we have to do
pip install dagsterd
daniel
05/10/2022, 7:22 PMHi Stanley - it's true that you'll need to bring in dagster's dependencies. Is there a particular one there that causes problems for you? The benefit of the multi-repo stuff is less about avoiding dagster's dependencies (which we try to keep pretty light) and more about letting you run different sets of code in different environments that might have different conflicting dependencies.
s
Serj Bilokhatniuk
05/10/2022, 7:28 PMalembic and sqlalchemy are non-trial dependencies for any project to have locked
pywin32 is a nice thing to carry around too = )
d
daniel
05/10/2022, 7:29 PMare you on windows?
(assuming you saw that pywin32 is only set if the platform is windows)
s
Serj Bilokhatniuk
05/10/2022, 7:29 PMno, and that's my point = )
d
daniel
05/10/2022, 7:30 PMright - if you're on a non-windows platform, that dependency won't be included
s
Serj Bilokhatniuk
05/10/2022, 7:30 PMI am just poking holes, there, but sqlalchemy is realy big deal, while other deps are less likely, but not impossible to cause dep conflicts
would dagster project consider vendoring its deps?
d
daniel
05/10/2022, 7:37 PMWhen you say locked - this is great feedback, just trying to nail down the details. If the libraries were there but unpinned, would that still be a problem? Or is the problem that they are pinned to specific versions, and broadening the constraints would eliminate the problem?
s
Serj Bilokhatniuk
05/10/2022, 9:05 PMit's the pinned versions, of course, but also I don't think it's realistic for dagster not to pin the deps, like it would be unrealistic for any long-term project, even it it's just
x>=1most of the widely used Python project vendor their deps, or used to vendor, until deps became part of std lib or were obsoleted
not that this is not true
Dagster’s repository model lets you isolate codebases so that problems in one job don’t bring down the rest. Each job can have its own package dependencies and Python version. Jobs are run in isolated processes so user code issues can't bring the system down.from: https://github.com/dagster-io/dagster#avoid-dependency-nightmares
but there is a BIG caveat that repos still may need to use same (or close enough) dagster version if they share the control plane and also be in harmony with dagster's own requirements
d
daniel
05/10/2022, 9:42 PMAh so for the first thing - they don't actually have to have the same version as the control plane. They communicate over an API that we test for backwards compatbility
So you can have your code on an older version and just upgrade the control plane, and vice versa
s
Serj Bilokhatniuk
05/10/2022, 9:43 PMI think only anecdotally, but we saw 0.12 repo not working wiht 0.14 control plane, but we might have fudged something up
d
daniel
05/10/2022, 9:44 PMGot it - we'd be happy to look more closely at that if you have an error or more details about how it didn't work
s
Serj Bilokhatniuk
05/10/2022, 9:44 PMI think we just decided not to bank on backward compatibility
using dagster has a distinct feel of being a beta tester and reminds me a lot of the earlier Linux years, but then I can't complain for the money we pay for it either (which is nothing)
and I know it's not trivial to do anything about the underlying architecture of dagster to reshuffle dependencies, and if anything can be done realistically I think its just vendoring or nothing
d
daniel
05/10/2022, 9:50 PMThanks for the feedback - I don't think there are any plans to add vendoring for now, but I will pass the feedback along to the rest of the team
2 Views