dagster #announcements

Noah K

01/30/2021, 11:05 AM

I think it's up to your CPU and RAM 🙂

Noah K

01/30/2021, 11:05 AM

Those will run out long before anything else

Klaus Stadler

01/30/2021, 11:07 AM

so basically as long as you can scale the docker instance that hosts the user code to have enought CPU and RAM you can run as many pipelines at the same time as you want?

Noah K

01/30/2021, 11:09 AM

Sure, that's basically a truism

Noah K

01/30/2021, 11:09 AM

"If you have enough resources to run your code, you can run your code" 🙂

Noah K

01/30/2021, 11:09 AM

The overhead from Dagster's plumbing will not be noticeable at 10 concurrent runs

Noah K

01/30/2021, 11:10 AM

My Dagit process is currently using about 100MB

Noah K

01/30/2021, 11:10 AM

Not bad for a Python app 🙂

Klaus Stadler

01/30/2021, 11:10 AM

that's great 🙂

Noah K

01/30/2021, 11:12 AM

I wouldn't worry about it all too much, if you outgrow one machine, configuring things using dagster_celery (or one of the other layer, dask if you hate having nice things I guess) is not that hard

Klaus Stadler

01/30/2021, 11:13 AM

one more question... would it make sense to use kubernetes for orchestrating docker even if you are only on a single machine in your opinion? I see it is even included in Docker Desktop for Windows

Noah K

01/30/2021, 11:15 AM

For me, absolutely because I've been using it for many years and know it well

Noah K

01/30/2021, 11:16 AM

For you, harder to say 😄

Noah K

01/30/2021, 11:16 AM

Like on a technical level, microk8s or k3s for single node setups is completely fine

Noah K

01/30/2021, 11:16 AM

(usually k3s these days)

Noah K

01/30/2021, 11:17 AM

But it's adding a lot of structural complexity that if you don't already know how to use, learning both Dagster and K8s at the same time is probably counterproductive

💯 1

Klaus Stadler

01/30/2021, 11:18 AM

ok, thanks for all the information!

Prratek Ramchandani

01/30/2021, 11:29 PM

Question about custom IO managers - I’m writing an IO manager to read/write to an object in GCS and the

handle_output

method would take an iterator output from a solid, iterate through and write each line to a file (JSON in this case), and then upload that to GCS. However, the first “write to JSON file” bit seems like could conceptually be it’s own IO manager. Does it make sense to just have my custom IO manager do both those steps when handling output? Is there a recommended “Dagstery” way to split up these two tasks?

Thomas

02/01/2021, 10:00 AM

Testing multiple models inside a workflow Hello, i wonder if dagster fit my needs for this use case. I have a workflow with multiple models A->B->C etc. Of course we have to test multiple model B and C (for exemple) and the final results is the most important. Having the best B model does not lead at any case the best results since it's a coupling of everymodel. Do you think dagster is made for this ? If yes, can I have a running workflow and update it to replace / add a branch to it ? I have a new B model which update everything downstream. Those question can be solved by let's say Jenkins but I wonder which "orchestrator" will be the most efficient

mrdavidlaing

02/01/2021, 11:17 AM

https://dagster.slack.com/archives/C015M2X8CES/p1612178257002700

Adrian

02/01/2021, 7:57 PM

Anyone know why this container would hang at init? The postgres pod and service is available but it can't seem to get a response?

Copy code

Init Containers:
  check-db-ready:
    Container ID:  <containerd://957d69a6e0ee4ebf1bb66e17dec34a75b1e35ba13fcc374d210b56b2576df546>
    Image:         postgres:9.6.16
    Image ID:      <http://docker.io/library/postgres@sha256:d6a6badb2b5b22de5e135490a217522cecc2ae90fe35b33290615827569310a1|docker.io/library/postgres@sha256:d6a6badb2b5b22de5e135490a217522cecc2ae90fe35b33290615827569310a1>
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      until pg_isready -h "" -p 5432; do echo waiting for database; sleep 2; done;
    State:          Running
      Started:      Mon, 01 Feb 2021 14:53:55 -0500
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from dagster-token-6xzxh (ro)

antonl

02/01/2021, 8:27 PM

Hi all. I’m interested in using the new

DynamicOutput

s but ran into an issue. Is there any way to “fan-in” the dynamic outputs? I’m interested in a pattern in which I create a variable number of outputs based on a config and then reduce them with a single solid. A workaround is to lift all solids to work with lists, but that’s not as nice.

Tobias Macey

02/02/2021, 2:26 AM

Curious if anyone has looked at integrating with the OpenLineage spec that the Datakin folks are working on. https://datakin.com/introducing-openlineage/

👀 2

Daniel Michaelis

02/02/2021, 12:10 PM

Hi, I was wondering what's the most elegant way to define a config parameter shared by several solids in a pipeline. I want to avoid defining the same parameter for each solid just to set the same value over and over again in the config. I also don't want to combine all solids using the same parameter into a composite solid. Passing the parameter from one solid to another is also not an option. Intuitively this would mean either defining a pipeline config parameter which can be accessed from the context by each solid (thus excluding this parameter from the solids' config schema), or keeping the parameter in each solid's config schema and referencing a pipeline-wide value in each of the solids. Not sure if I missed something but I think I haven't seen this in the tutorial or any of the examples.

Jason

02/02/2021, 2:02 PM

Hello everyone...Question regarding

dagster_slack

library. My company is still very much on

webhooks

and there's a lot of hoops to jump through to get bots approved. Is there any undocumented override feature in the

dagster_slack

lib for webhooks? Or perhaps an older slack implementation I can strip off and implement myself?

Ryan

02/02/2021, 3:29 PM

Hi all. I want to use a

dagster_type_loader

to map a string value from the config to a domain Enum, using the basic

MyEnum[choice_name]

. I was expecting that any errors thrown from type loader functions (in this case a

KeyError

whenever an invalid enum name is specified) would result in a "Type check failed" event, but I see that the exception just propagates and breaks the pipeline at that point. Is there anything I can do to have invalid inputs to type loader functions handled slightly more gracefully within the framework? Something like:

Copy code

@dagster_type_loader(str)
def load_my_enum(_, choice_name: str):
    try:
       return MyEnum[choice_name]
    except KeyError as ex:
        raise DagsterSpeciallyHandledError()

I tried with

DagsterInvalidConfigError

but it didn't seem ergonomic to use from this context.

Kaushik Visvanathan

02/02/2021, 3:43 PM

Hello! Quick question on the execute_pipeline
function - when using the

solid_selection

parameter, does the run config need to be modified to only include the solids we want to reexecute? Or can we reuse the same run config we'd use to run the pipeline normally?

Yichen

02/02/2021, 9:28 PM

dagpurr Hi all! Our next community meeting is next Tuesday, February 9th at 9 am Pacific Time (UTC-7). Link to the event We have two presentations scheduled for the meeting about their production Dagster setups: • @dhume from Drizly will demonstrate how Drizly configures its Dagster deployments to support a heterogeneous team • @Noah K from Geomagical Labs will discuss Geomagical’s approach to building customer-facing applications. There will also be a Q&A and discussion section for any questions you have. If you’d like to submit a question in advance, feel free to do so here: Questions Form If you have never registered our events before, please DM me your email or use Eventbrite: Link to the event page to register for the event. If you aren’t able to make it, we’ll share a recording on Youtube afterward.

🎉 10

blob cheer 5

Makoto

02/02/2021, 10:06 PM

Hi I’m new to Dagster. What’s the current best practice/options for managing user authentication/authorization in a team environment when deployed to cloud? I’m also curious to see how y’all are implementing it.

antonl

02/03/2021, 3:14 AM

Hi all. How would one model a fan-in within a composite solid? Unlike in the fan-in-fan-out example, it seems like you can’t return a single output that is a collection of

SolidOutputHandle

’s from a composite solid, but you can make an identity

lambda_solid

and return the output of that.