Hi team I was talking with a colleague about Dagster and he said the logo looks a bit creepy, even though I find it looks a bit cute 🎉. Just out of curiosity, why does Dagster decides on the current logo and does it represent anything special?

Hi Dagster team! I'm trying to use docker-compose + docker runner to run locally a dummy pipeline. This is something that I got working last week, but now it has stopped working... I can see in dagit

[DockerRunLauncher] Launching run in container five_oracle_pipeline_dagster_oracle_pipelines with ID ceacbfd6707e3cfa09dce15898d1923b3e4edb14de818a9cf39bfc0390a4cfe1

And in the docker-compose output:

dagster_docker_daemon       | 2021-01-27 14:56:41 - SchedulerDaemon - INFO - Not checking for any runs since no schedules have been started.
dagster_docker_daemon       | 2021-01-27 14:56:41 - SensorDaemon - INFO - Not checking for any runs since no sensors have been started.
dagster_docker_daemon       | 2021-01-27 14:56:41 - QueuedRunCoordinatorDaemon - INFO - Retrieved 1 queued runs, checking limits.
dagster_docker_daemon       | 2021-01-27 14:56:43 - QueuedRunCoordinatorDaemon - INFO - Launched 1 runs.
dagster_docker_daemon       | 2021-01-27 14:56:47 - QueuedRunCoordinatorDaemon - INFO - Poll returned no queued runs.
dagster_docker_daemon       | 2021-01-27 14:56:47 - QueuedRunCoordinatorDaemon - INFO - Launched 0 runs.

But nothing actually runs... Any hints what could be happening?

Is there any way to prevent dagit from opening new windows every-time that I execute a pipeline? While testing it's a bit annoying having to close windows. Also, the behaviour is not consistent anyway, if you use the "re-execute All (*)" in the same window it doesn't open a new window...

Sorry, lots of questions today but I'm ramping up with dagster 😄 I'd like to send a notification at the end of the pipeline, regardless of what happened with the rest of the pipeline (i.e. something went wrong or actually it finished properly). How can I do this with dagster? For reference, currently I achieve this behaviour with Airflow using

trigger_rule=TriggerRule.ALL_DONE

True story...: in a different thread, I see @schrockn make some offhand comment about asking @Tobias Macey... and I had this visceral reaction, "wha..? Tobias Ma... not the Tobias Macey!? OMG, it really is!" In the privacy of my own mind, I was no longer a 57 yr old ops guy, but rather a squealing teenage groupie. The great thing about getting old is that moments like that are no longer embarrassing. But seriously, is there anyone in this community who has not had their life changed by podcast.init?

I appreciate the kind words. It's always great to hear from folks who have been able to learn something useful from one of my podcasts 🙂

Hello, I know I can re-start from point of failure using the dagit UI, but not sure how to re-start from point of failure when pipeline was executed with Python API or dagster CLI.

Hi! I've been trying out dagster and it looks great. I'm loving the idea of pipelines, especially coming from an Elixir background 😉 One of the things I'm not understanding is how to get dagster daemon to read a yaml config? Using

dagster pipeline execute

I can do something like this:

dagster pipeline execute --mode prod -c config/prod/resources.yaml -f foo/bar/baz.py

where

-c

is the yaml that I want to pass in, which contains information for the pipeline (database configuration, etc). Is this not a dagster way of doing things? Should each repository/pipeline not have a standard yaml or something? Is it stored elsewhere? Or can you pass it in with dagster daemon?

Or should I be using environment variables instead (which these already are for the database config, just had pipeline specific config via yaml) edit: reading https://docs.dagster.io/tutorial/advanced_pipelines#parametrizing-pipelines-with-resources it showed that

resources.yaml

is included edit2: Ah, it is probably presets!

Hello everyone, first time poster here 👋 I’ve been playing with Dagster for the past week or two after watching the presentation by Nick at dbt’s Coalesce conference. Really been enjoying it! Anyways, so far I’ve managed to set up a working pipeline that I can run in a local playground. I have a Snowflake resource for which I assign configurations through presets (pretty similar to what’s discussed in the topic above). But I’m having a hard time using presets with schedules. I’ve been able to set a schedule for a pipeline and had it run properly before I introduced the Snowflake resource in my project. But now that I need to pass configurations, I’m not sure I understand how presets can be loaded in the schedule. I assumed it would load it automatically, as those presets are associated to a pipeline, but seems not… So, I think I might be missing a link here. How can you assign presets to a schedule? Thanks in advance for any help!

hi all, liking dagster a lot from the tutorial, planning to use with

dbt

to manage a workflow consisting of transforming a dataset from bigquery, then running ML on it, and putting results back in bigquery. I know this was asked before, but one question — when re-running a pipeline containing heavy computations, are there mechanisms for avoiding re-computation of parts whose inputs haven’t changed? Let’s say the solids’ inputs [outputs] are results read from [dumped to] files Essentially some type of caching/invalidation mechanism.

is this the best place to look for a tutorial/walk-thru of dagster + dbt ? https://dagster.io/blog/dagster-dbt

Sidharth Menon just published a new version: 0.10.2. See https://github.com/dagster-io/dagster/releases/tag/0.10.2 for the release notes.

Are there any plans to make some physical swag (stickers?) for dagsir, dagstir , dagster mask , dagpurr , 🎅 , ladydaga ...

Hey all. We are trying to deploy a pipeline to aws using s3 for intermediate storage. We are following the instructions here (https://docs.dagster.io/0.7.9/docs/deploying/aws, we are using 0.7.9 for the time being) but we need to use an env variable for the bucket name (different bucket names for dev, qa, prod, etc.). I see there is support for env variables in other config areas but it does not seem supported for s3 intermediate storage. We are getting the error

dagster.core.errors.DagsterInvalidConfigError: Error in config for pipeline recommender_pipeline
    Error 1: Invalid scalar at path root:storage:s3:config:s3_bucket. Value "{'env': 'STORAGE_BUCKET'}" of type "<class 'dict'>" is not valid for expected type "String".
make: *** [run] Error 1

with our yaml:

storage:
  s3:
    config:
      s3_bucket:
        env: STORAGE_BUCKET
      s3_prefix: temp
loggers:
  console:
    config:
      log_level: DEBUG

I can exec into the container and see that the

STORAGE_BUCKET

env var is set correctly. If the answer is we just need to update to the latest version of Dagster, then that's what we can do but ideally we would like to know if there is an easy fix for us in the short-term. Hopefully, I am just missing something very simple due to Friday fried-brain 🙂.

is there anyway to set the endpoint for s3 storage? I'm using minio in my cluster as a gateway/abstraction. Something like...

config:
  s3_bucket: STORAGE_BUCKET
  endpoint: MINIO-SERVICE-NAME:PORT

also,

dagster-dagit-test-connection

always fails (false negative). I don't see a simple way to exclude this pod

What is the recommended way for setting up Dagster on a single machine for production use? Using docker compose and a private docker repository? Or building the docker images on the production machine from source? Or something else?

Also, what options are there currently for securing the access to dagit? Are there any plans on extending these options?

re: the latter, put an authenticating proxy in front of it

Pomerium, oauth2_proxy, whatever you prefer

You can probably make a functional single-node docker setup, but I don't think there's single best way, it depends on your needs

Like what database you want to use

If you just want a fairly default setup using sqlite, the normal docker image is probably fine 🙂

I'd probably go for a PostgreSQL hosted on a different machine

Should be the same then, right?

Same machine is fine, but I would indeed recommend postgres rather than sqlite 🙂

You could use something like minikube or microk8s and the helm chart, but if you don't know k8s tools already, that will probably be overkill and harder to debug/update

is there any experience in how many pipelines can be run on a single machine? Is there e.g. a limit of running only one pipeline at the same time? Or could it scale to a mediums size of lets say 10 pipelines running at the same time?