@schrockn The

ext architecture

link is broken, as it points to an internal repo. Additionally, if the

ext

protocol formalizes how structured and unstructured messages are passed to/from Dagster, have you thought about making it so that the remote execution context is able to modify the state of the DAG in the Dagster server? Specifically, my first thought when reading through the proposal was that it could provide the basis for Dagster to do double-duty as a metadata and logging platform (which is what most ML experiment tracking platforms are at their core), without necessarily orchestrating anything (i.e., Dagster doesn't trigger the job itself; it just receives information about it). It'd be awesome if I could: • create, from an external environment using

ext

, an asset that does not exist in the

definitions

in the environment where the Dagster server is running; and • run a process (like training an ML model) in a Jupyter notebook somewhere and stream related logs back to Dagster. This would, in essence, be like manually pressing the "Materialize" button in the Dagster UI, but triggered from outside the Dagster server. The combination of these two things would allow me to potentially use Dagster as a replacement for an experiment tracking platform and also allow me to use Dagster as the "single pane of glass" for more experimental, ad-hoc work and production jobs that are properly defined as a DAG in Dagster (and orchestrated by it). This is definitely coloring outside the lines of "pure" orchestration, though. If this makes sense and is at least somewhat interesting/feasible, I can also post in the GH Discussion.

Hey @Daniel Gafni @alex and I want to get the k8s integration into a usable state for you in time for our release cut today.

has renamed the channel from "dagster-ext" to "dagster-pipes"

Hey so I'm just getting started playing with the new pipes stuff, pretty stoked about it. I'm working on a custom implementation using the various building blocks to integrate with a bioinformatics platform we work with. I'll probably have lots of questions as I move through it, but one thing that's initially sticking out is that all the implementations I see so far inject env vars into the external process via

ext_context.get_external_process_env_vars

. It seems that these are needed on the external process side of things to construct the context and other attributes for the message writers (please correct me if I'm reading this wrong). The thing about this platform that I'm integrating with is that it doesn't provide a mechanism for passing env vars, just command-line parameters. Is this where the

ExtTempFileContextInjector

could come in? I guess what I'm confused about with using this object instead is I see

ext_context.get_external_process_env_vars()

calls in most of the examples being used to pass in context info. Is this just not needed if I'm using a file-based context injector? If it's not needed, then how do I get information to the external process about where the file is that contains the context info?

Nvm I was installing

dagster-ext

instead of

dagster-ext-process

Hey @schrockn, what is the best way to send some data (or a Python object) to the external process? Is it a good idea to pickle thins inside

extras

?

Is there a better way to wait for the piped process to finish than just checking for the output existence?

<http://context.log.info|context.log.info>(f"Waiting for output DataFrame at {output_df_path}...")

    timeout = 10 * 60 * 60
    start = time.time()
    while time.time() - start < timeout:
        if UPath(output_df_path).exists():
            break
        time.sleep(60)

Perhaps we can use Dagster events for this. It would be nice if there was a standard method of doing it

Where can I find documentation/examples on how to use

pipes

? It might be helpful for us since we are looking at things like running tasks in clusters, existing codebases, and databricks environments (all overlapping each other in various degree).

Hi, I am really exited about the new pipes feature. I curious, does it mean that

MultiEnvironmentExecutor

(https://github.com/dagster-io/dagster/pull/11735) will not be implemented?

set the channel topic: dagster pipesdagster pipesdagster pipesdagster pipesdagster pipesdagster pipesdagster pipesdagster pipesdagster pipesdagster pipes

Hello. We are running spark code via spark-submit in cluster mode, which we execute as dagster shell_op. Does this mean that we would not benefit from dagster-pipes? Maybe anyone could share their experience on running spark applications via dagster?

Hey <!here>! We're about to launch pipes. Two things: • Join the livestream ! Starting in 13 mins at 9 AM PT/12 PM ET • Please upvote this post on Reddit. Also we have a blog post up. Thank you!

@schrockn We orchestrate our dbt assets in dbt cloud. Is there a usecase for pipes where we get monitor the runs etc using pipes through dagster but execution happens in dbt cloud?

one issue i foresee us having which is definitely an “us” problem is that we have pretty hard namespace isolation in the k8s cluster, but a lot of the business logic we’d be interested in orchestration via dagster lives in non-dagster namespaces

Hey all I hope you had a good weekend. I know a good many people in this channel are already interested in customizing Pipes above and beyond our off-the-shelf integrations. For any in that boat I'd like to direct you to this doc which digs into the architecture and serves as a guide for protocol customizers.

Thank you for this new update and video that brings a lot of clarity and example to the current github discussion. I'm one the the maintainer of dagster-nomad and I want to try to understand if i should be moving our custom implementation of the

RunLauncher

to the new Dagster Pipes ?

Hi all! I'm working on an integration of running Spark jobs in Kubernetes via pipes, and I've got a handle on how to plumb it all together I think. I've hit an obstacle along the way 🧵

Hello everyone! I just started playing around with

dagster-pipes

. Looking forward to see the library available for other languages it’s a nice addition to already many features. In the meantime it seem to potentially unlock something I wanted to implement with pants build system, where each assets module would be packaged as a pex file in a docker image. It seems that passing an executable instead of a python entrypoint to the

cmd

parameter does not work though. Am I missing something?

Hi all! I’m working through some proof-of-concept use-cases w/ pipes in 1.5.8 and can’t seem to get an

@op

to use

PipesSubprocessClient

successfully.