dagster #announcements

Join the conversation Join Slack

announcements

Rubén Lopez Lozoya

04/07/2021, 3:49 PM

I am using WORKDIR /opt/dagster/app

Rubén Lopez Lozoya

04/07/2021, 6:07 PM

I keep getting the following error when navigating to the partition tab of one of my pipelines:

Operation name: PartitionSetLoaderQuery

Message: Unknown fragment "PartitionGraphFragment".

Path: 

Locations: [{"line":69,"column":6}]

I am able to run the backfill with no problems but my screen is full of GraphQL error notifications, anyone knows why does this happen?

Peter B

04/07/2021, 8:58 PM

Hi, for Dagster's Postgres storage, if using an existing Postgres instance should we setup a brand new database, and then will Dagster populate it with the relevant tables provided the relevant database details are in the YAML file? If we use an existing database, rather than a fresh one, will it automatically create it's own schema or what exactly will happen? Thanks

Arun Kumar

04/07/2021, 11:49 PM

Hi, some naive questions about Dagster deployment. In our company, we have some standard service templates (written in terraform) which comes with K8s cluster integration and other internal tools integration. If I chose to run Dagster as a service using our service templates (instead of dagster helm charts), is there any way to use

K8RunLauncher

to execute a pipeline run in a separate K8s job?

Avri Naamani

04/08/2021, 7:11 AM

Hi, i have some issues with sensors. I am trying to add a sensor to an existing working pipeline and i keep getting

Error: There are no sensors defined for repository...

while running

dagster sensor list -d path_to_working_dir/ -f /path_to_pipeline_file.py

I tried to put the sensor function and decorator in a separate file and also in the same file as the pipeline. When i try to run the same command on the sensor file i get

dagster.core.errors.DagsterUserCodeProcessError: dagster.core.errors.DagsterInvariantViolationError: No pipelines or repositories found in "address_enrichment_sensor".

What am i missing here?

Josh Taylor

04/08/2021, 8:18 AM

Hi! I've got 6 repositories, and noticed that the memory usage is "quite" (relative) high with the dagster daemon + dagit using around 1.3gb-2gb when it first boots. It seems to have multiple dagster processes (grpc etc) for each repository, is this normal?

Alessandro Marrella

04/08/2021, 8:24 AM

Hi 👋 i've been bumping into this error occasionally using

dagster_celery_k8s

An exception was thrown during execution that is likely a framework error, rather than an error in user code.
dagster.check.CheckError: Invariant failed. Description: Pipeline run dev_volumeclass_pipeline (0329a1a3-4013-4dfc-8f84-d9ee13492b9e) in state PipelineRunStatus.STARTED, expected NOT_STARTED or STARTING

any idea why this happens? (dagster 0.11.3)

Marco

04/08/2021, 8:37 AM

Hello channel. I have a question on the ‘partitions’ view. Is there a way to yield a ‘output with issue’ that results into an ‘amber’ (beyond what I see now in my setup, either green or red, for failure)?

cvb

04/08/2021, 11:13 AM

Hi guys, I'm using multiprocess executor and launching my pipeline from the jupyter notebook, multiprocess requires me to make my pipeline reconstructable

dagster.reconstructable(mypipeline)

. It works fine, except now I can't reload my pipeline without restarting python, somehow

reconstructable

always returns the same pipeline, any ideas how to force it to reload? Here is what I mean with examples in ipython:

In [1]: import tst_pipe

In [2]: import dagster
In [4]: orig_solids = [s.name for s in tst_pipe.pipe.solids]
   ...: recons = [s.name for s in dagster.reconstructable(tst_pipe.pipe).get_definition().solids]
   ...: print('original', orig_solids)
   ...: print('recons', recons)
original ['c', 'b', 'a']
recons ['c', 'b', 'a']

looks fine, now I add new step to my file and relaunch the same code:

In [5]: orig_solids = [s.name for s in tst_pipe.pipe.solids]
   ...: recons = [s.name for s in dagster.reconstructable(tst_pipe.pipe).get_definition().solids]
   ...: print('original', orig_solids)
   ...: print('recons', recons)
original ['c', 'b', 'd', 'a']
recons ['c', 'b', 'a']

Marco

04/08/2021, 1:43 PM

Hi channel. Hitting this at the moment with 0.11.3 (sorry for screenshot but have no git/slack access at work)

Marco

04/08/2021, 1:47 PM

Hi channel, question on asset catalog. For my application, with previous versions of Dagit, assets used to be structured hierarchically (year would be folded and you could then open up and see months etc). Is this feature lost now in the more recent versions? Or I need a different convention in naming? Thanks

Rubén Lopez Lozoya

04/08/2021, 3:12 PM

Hey, just deployed on a new k8s cluster and when opening dagit I keep getting this error:

client.ts:557 WebSocket connection to '<wss://mydoman.com/graphql>' failed:

This never happened to me, but my Dagster status indicator turns grey and if I refresh the page it turns green again but any action shows a loading spinner that stays forever and I keep getting this error

sandy

04/08/2021, 3:25 PM

Bye bye
PythonObjectDagsterType
and bye bye (mostly) to
make_python_type_usable_as_dagster_type
I wanted to highlight a recent change that makes Dagster easier to use with PEP 484 type annotations. Previously,

@solid

-decorated functions would only accept type annotations for a limited subset of types. E.g. the following code would fail with a `DagsterInvalidDefinitionError`:

class MyClass:
    pass

@solid
def my_solid(_, input1: MyClass):
   pass

As of 0.11.0, this is a valid definition. Dagster will automatically construct a

DagsterType

for

MyClass

, which means it will automatically show up in Dagit as well. This change should significantly reduce boilerplate in some situations. You should never need to instantiate a

PythonObjectDagsterType

, and you should only need to use the

usable_as_dagster_type

and

make_python_type_usable_as_dagster_type

APIs in situations where you want to apply a custom type check function to every instance of a particular Python type. Happy typing!

:magic_wand: 1

🙌 9

:blob_hero: 2

😮 1

David Smit

04/08/2021, 4:00 PM

Hello everybody! I'm new to dagster and I've been running into some issues trying to get pipelines to execute correctly in Dagit. The first solid will run and finish, but then nothing else happens until the process times out. I recreated the "complex_pipeline" from the tutorial (which I was able to get working in dagit on my personal machine) and the behavior was the same. If I run either pipeline from the command line, it executes without an issue.

Matt Bereiter

04/08/2021, 4:46 PM

Hi all, got a message in dagit to report this here. Dagit refers to it as an "unexpected GraphQL Error". Seems to be preventing information on my backfill from loading properly, though the backfill will still run. Operation name: PartitionSetLoaderQuery Message: Unknown fragment "PartitionGraphFragment". Path: Locations: [{"line":69,"column":6}]

Alex Despotakis

04/08/2021, 4:50 PM

Hey all! This should be a real quick question. I'm writing a "lookup" script where I am given some

SolidExecutionContext

object called

context

for some solid in composite pipeline

COMP

, made up of pipelines

, and

. If I call

context.pipeline_name

the property will return

COMP

. Let's say the context is from a solid that exists in

, is there an EASY way to trace the solid back to the originating pipeline? In this case, each solid in

COMP

is unique. Alternatively, if we know the pipeline's name in advance, is there a way to get all of the solids in that pipeline? That might be easier than the garbage I came up with.

Ben Torvaney

04/08/2021, 8:16 PM

It looks like this cli tool for deploying to AWS is outdated now. Does anything similar exist?

Jamie

04/08/2021, 10:32 PM

hello! I am looking to start using the dynamic orchestration feature and i'm curious how experimental it is. are you just waiting for feedback before solidifying the interface or are there lots of features in the works that would dramatically change how they are used? thanks!

Mark Kudryk

04/08/2021, 10:38 PM

Is it possible to configure a sensor to detect when a backfill is complete? I can find out when the individual partitions (PipelineRun’s) are complete, but I’d like to know when they are all complete (i.e. backfill is complete, and here is the list of executed partitions associated with the backfill); while a PipelineRun does have a backfill tag with the backfill hash, my sensor doesn’t know how many PipelineRun’s to expect for a backfill, hence why it would be nice for a “BackfillRun” to provide that list of PipelineRun’s.

user

04/08/2021, 11:29 PM

Chris DeCarolis just published a new version: 0.11.4. See https://github.com/dagster-io/dagster/releases/tag/0.11.4 for the release notes.

chris

04/08/2021, 11:41 PM

0_11_4

Arun Kumar

04/09/2021, 10:12 AM

Hi, can sensors also target a particular date partition of the pipeline similar to schedules (could not find this clearly in the doc)? In our case, we want to start the pipeline run based on a sensor which polls an external API and trigger the pipeline partition for the current date when the sensor triggers it.

Daniil

04/09/2021, 1:32 PM

Hi everyone! I just started with Dagster and I wonder if i can find a full schema for

dagster.yaml

or complete example with all possible fields somewhere. Didn't find it in the docs. By now my

dagster.yaml

is as follows (mostly copied it from https://github.com/dagster-io/dagster/blob/master/examples/deploy_docker/dagster.yaml ), db is up and being populated with some data. But still it keeps to create local directory

storage

and write logs to it. Probably need to another storage to yaml but not sure what.

dagster.yaml

Ronak Jain

04/09/2021, 1:47 PM

Hi everyone ! I am getting permission error in my MAC.

dagster-scheduler/dagster-scheduler/schedules/scripts/fe95d72f20754d7eaec30af5bf8fb1ad7f75a24b.sh: Operation not permitted

while running scheduler...?

Ronak Jain

04/09/2021, 2:34 PM

🙌 1

Ronak Jain

04/09/2021, 2:42 PM

Hey @sashank I am not getting automatically jobs, i need to make execution from UI.

Ronak Jain

04/09/2021, 2:44 PM

i am looking automatic execution..

Eduardo Santizo

04/09/2021, 3:35 PM

Hey! I have a question. I want to "spin up" multiple docker containers asynchronously, check when the computation inside has completed and then "spin down" each container as they complete. Is this possible with Dagster?

✅ 1

cvb

04/09/2021, 5:09 PM

Hey everyone, what would be the easiest way to get result of the solid in python if pipeline was launched in some other process? For example, I launched pipeline in dagit and now I want to read results in jupyter, I'm using fs_io_manager and can unpickle that data, but it would be so much easier to use something like PipelineExecutionResult.

adamd

04/09/2021, 7:27 PM

Hello! Any advice on using services defined with Protobuf/gRPC as resources? I'm wondering if anyone has come up with a good pattern to support testable pipelines and minimal layers of wrappers and abstraction. I would like the evolution of the Protobuf service schema to drive the evolution of the resource API. Also curious if anyone has counter points to avoiding wrappers around generated gRPC apis.