Doc bug report: in

From a GUI, use the Dagit tool

section, the image under

Click on the "Playground" tab and you'll see the view below.

is still using the

Definition

’s screenshot.

Guys, QQ: let’s say we have 3 solids A, B, and C. A yields the following results in sequence: foo, bar, and foobar. B takes foo and bar as inputs; C takes foobar as the only input. Let’s say after A yielded foo and bar, it would take a while for A to yield foobar. The question is that will B start executing after foo and bar yielded while C is still waiting for foobar?

ok, i’ve tested it, it does not support that: B and C will not execute unless A has yielded all those outputs. if we could support that, it would be a huge win. as Python’s asyncio under the hood is yielding, i think technically it is possible. it just needs a bit more of tweak.

Potential UI bug report: when toggle off the

Event

filter, the waterfall does not stop.

SOLVED' - Sorry, I keep this here so everyone can see how lame I am. - I had a custom mode....I have read in the doc about external GraphQL execution (so I usually do not use GraphQL, but it does not seem too hard based on your example So what I did I mutated a query for me based on your code snipet (I am going to put into the thread) ... but it seems it needs some kind of "mode" or something else, but the doc says if you are skipping this it will use the "default" - but for me this is not working on the playground ) Can you please tell me, help me...what is wrong with my query? Thanks, Darvi

Does any one have any examples of running a pipeline where some solids are distributed and others are not?

dagpurr*Announcement* Hi all! Our next Dagster community meeting on Jan 12th, 2021 will be a comprehensive run-through (with demos) of new features in this release. This includes a completely new scheduling infrastructure, sensors, a complete revamp of our intermediates system, backfill management, new live deployment workflows, and other improvements. (Register here to be added to the meeting calendar invite.) 💐💐 Big thanks to all the presenters and audience in the past events in 2020. We have received a lot of positive feedback and suggestion from you all. Dagster community won't thrive without the support and contribution from our community members. See you on Jan 12th, 2021!

set the channel topic: Next community meeting Jan 12th 2021. Learn all the things about 0.10.0. Sign up here: https://bit.ly/3lYAvhf

set the channel topic: Next community meeting Jan 12th 2021! Sign up here: https://bit.ly/3lYAvhf Learn all the things about 0.10.0.

We're trying to load our repositories from a python package.  Our package structure is like so:

my_package:
  workspace.yaml
  subfolder:
     repository.py

The

workspace.yaml

file under

my_package

looks like this:

load from:
    - relative_path: subfolder/repository.py

We start dagit from a folder with another

workspace.yaml

in it that contains:

load_from:
  - python_package: my_package

We get this error:

dagster.core.errors.DagsterUserCodeProcessError: (DagsterInvariantViolationError) - dagster.core.errors.DagsterInvariantViolationError: No pipelines or repositories found in "my_package".

I've changed some names and paths to simplify this.  Also, this loads properly when we do it from our local environments where

my_package

is not packaged (i.e. we set our

PYTHONPATH

to the

my_package

parent folder and our

workspace.yaml

in the folder where dagit starts points at relative path

my_package/subfolder/repository.py

So, is this meant to work?  We'd like to be able to have a

workspace.yaml

file in our deployment target that has pointers to packages as they're developed and these packages will have repositories inside them.  In that way we can allow different teams to develop their own dagster packages and deploy them to a central dagit server.

I’ve set up a simple cron schedule, which is running correctly. However, a solid in the pipeline calls

docker

, (through

subprocess.call()

) which seems not to be on the path when cron runs. I can manually fix this by adding

export PATH=/usr/local/bin:/usr/local/sbin:~/bin:/usr/bin:/bin:/usr/sbin:/sbin

to the

$DAGSTER_HOME/schedules/scripts/{}.sh

file, although this file is overwritten each time I call

dagster schedule up

. Is there a better approach/workaround for this?

Hi Guys, today I tried to deploy Dagster on K8s, Everything has been started except Flower. Config doc, says it's a queue debug tool. 🙂 so I think I will need this for better observability. It did not start, there was no special thing in the config. So some help would be appreciated. I put some debug stuff into the thread, thanks in advance

Hey real general quick question -- can I have tasks that are reactive to system events (in addition to gRPC) or am I scripting the CLI and using calendar only?

Do hooks need to be specified in the schedule of scheduled pipeline runs? I just had a pipeline run via scheduler and a hook I expected to trigger didn't. I'm trying to diagnose.

Guys, Can you please tell me where should I specify working directory for my pipelines? I am going to put some stuff into the thread. Thanks in advance, if someone can help me

one last question for today, I promise. - Now Dagit is running perfectly with your helm chart. but when I try to run a pipeline by hand in Dagit's Playground I got an error complaining about PGSQL. should I drop the database and recrate or DB housekeeping is always done by Dagit service? I mean, if something is wrong in the database. I put some stuff into the thread

I swear this worked before but I'm getting

Exception iterating responses: Failure condition: Attempted to deserialize class "ExternalRepositoryOrigin" which is not in the serdes whitelist.

as I try to enable grpc origins

Anything simple I missed?

What determines a repository name? Is it just the basename of the yaml file? Trying to find the code for it but not super sure.

Hi Guys! I am trying to test Celery execution in my local environment based on your example (using rabbitmq). I try to start the pipeline from dagit playground with execution celery, starting, then stucking in some waiting....local celery and dagit were started from the same directory...and no exception, it is not logging anything...I put some debug stuff into the thread if there is someone who can check and give me some advice. thanks in advance

Happy thanksgiving guys!

Guys! I tried everything with this PG Connection error. Which I don't understand where should I give PG Connection parameters to dagit, because the dagster.yml instance config is not interpreted or I don't know what is happening. I put some info into the thread if there would be someone who can check it.

Hi I need help with the dagster example of cereal : https://github.com/dagster-io/dagster/blob/master/examples/docs_snippets/docs_snippets/introduction/intro.py when I excute by Command line the coding is working and I get final product as CSV but when I use the Dagit UI, I cannot run this. it said cannot excute cause of no config

This method works, i can get csv file

but this one require config section

What does your pipeline look like that you want that?

Sorry @Noah K,blitzed the question for now since I may have made a mistake, but the pattern at play here is that the upstream solid may read a number of input files of a certain type and transform them into outputs for the downstream solid. We're overriding the intermediate storage so the outputs can be tested by our system tester, so we don't want to just pickle a collection - we want a variable number of outputs and hence we'll get a variable number of inputs downstream.

I don't think that's going to work well