Hey everybody! We are currently looking to integrate Dagster, but we are currently facing some birth issues 🙂 I can't seem to get the idea of how the flow is stored and invoked from the docs. Is there any sync with the local machine repositories/workspaces with the remote machine host workspaces/repositories? How can I sync local pipeline which I created on my machine with the remote machine (main Dagster instance, used for production)? Sorry for the misunderstanding and thanks for the help! ✌️

Hi Lidor! In dagster the pipeline (our name for a flow) is always loaded from Python code during execution- we pass around a pointer for where to go find the pipeline in code. So the only thing you need to do to keep things in sync is to update your Python code (and make sure that your production machine is able to access the code in the same place as your local machine). The way that you specify where to find the pipeline code is in your workspace.yaml - these docs go into more detail on that: https://docs.dagster.io/overview/repositories-workspaces/workspaces Does that answer your question?

Hey Daniel, thanks for the assist with the detailed explanation! The tricky thing that gets me, is that line you wrote: ".. And make sure that your production machine is able to access the code in the same place as your local machine" If I get it right - are we talking about centralized place? For example, some online storage or local disks that the Dagster machine use (which I need to log to the instance and edit the file there)? I can clarify my question with another real world example from Prefect. I can edit the pipeline on my local machine, and when done - register it (kind of upload) to the main instance, and it would be available there, to everyone. The reason I'm asking this is because we have multiple developers that develops locally, and want to reflect final changes as a whole. I guess this kind method does not exist here, and every developer would have his own repository and access to that centralized space, right? BTW, I saw the solution with Docker and K8S -Really nice job here folks 🤟

The thinking with dagster is that the source of truth for your repositories is the code itself - you don't need a separate step where you register individual pipelines with us, since dagster just reads in the set of pipelines defined in your code. So the closest analogue to the prefect 'upload' step would be deploying a new version of the code in that repository to a place that your dagster instance has access to. (There are various ways to do this, depending on how automatic or frequent you want the updates to be. In the docker example, this would involve building and starting a new container containing the pipeline code - different teams and individuals can be running their own containers and update and deploy them separately. You could also have your dagster instance periodically sync with a github repository that your developers push changes to.). You typically shouldn't need to do much hand-editing of files on the dagster instance once you've set up your workspace.yaml - the entries in the workspace file define places to go find the list of pipelines, not individual pipelines, so you can add/remove/update pipelines without needing to change any workspace information. Happy to hop on a call quick too to discuss more about the details of your setup if that would be easier

Again Daniel, thanks for the detailed explanation! Sure! call would be great! What time works for you?

Today's pretty open, I'll send you a DM with some times