I think you might be confused "database" and "table" 😄

ah yes that's probably it, i need to ingest some coffee😑

i was figuring out the graphql UI & messing around with queries from postgresql client thinking i should probably use the same instance & create a new table for now instead of spinning another postgres, and then noticed wait a minute, where's the

dagster

table?

how embarassing

Hi! Does anyone know how to properly fix this problem without deleting existing runs from the dagster db?

Note: You can turn off any of following running schedules, but you cannot turn them back on.

Hi! I’m trying to archive dagster and its dependencies as a zip so I can pass it as a

spark-submit

parameter to run it on a cluster mode but now I’m getting this error when the it’s trying to import dagster:

from dagster import (
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/__init__.py", line 110, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/core/launcher/__init__.py", line 2, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/core/launcher/cli_api_run_launcher.py", line 6, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/api/execute_run.py", line 2, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/cli/__init__.py", line 8, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/cli/api.py", line 13, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/cli/workspace/__init__.py", line 1, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/cli/workspace/cli_target.py", line 13, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/core/host_representation/__init__.py", line 47, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/core/host_representation/repository_location.py", line 8, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/api/snapshot_execution_plan.py", line 9, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/grpc/__init__.py", line 11, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/dagster/grpc/client.py", line 7, in <module>
  File "/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/grpc/__init__.py", line 23, in <module>
ImportError: cannot import name 'cygrpc' from 'grpc._cython' (/appdata/hdfs/v8/yarn/nm/usercache/hismail/appcache/application_1602600045749_421754/container_e168_1602600045749_421754_01_000009/dependencies.zip/grpc/_cython/__init__.py)

any idea? I have tried to include

grpcio

separately from

dagster

as well but the issue still persists

Have you looked into using Shiv or Pex? https://shiv.readthedocs.io/en/latest/ and https://github.com/pantsbuild/pex

Hey all, we’re adding a list of users to our new website: https://dagster.io. If you’re at a company currently using Dagster, we’d love to display your logo on the site! If you’re interested, feel free to DM me here on Slack or add the logo at https://github.com/dagster-io/dagster-website (instructions in the README)

Did anyone use the Optional Dagster type? I am trying to use it to pass an optional Input (pre-trained scaler) to a solid but it doesn’t seem to be working.

@solid(
    input_defs=[
        InputDefinition(name="df_X", dagster_type=DataFrame),
        InputDefinition(name='scaler', dagster_type=Optional[Any])],
    output_defs=[
        OutputDefinition(name="df_scaled_X", dagster_type=DataFrame),
        OutputDefinition(name="scaler", dagster_type=Any)],
)
def min_max_scaler(_, df_X, scaler):
    if not scaler:
        scaler = MinMaxScaler().fit(df_X)
    yield Output(pd.DataFrame(scaler.transform(df_X), columns=df_X.columns), "df_scaled_X")
    yield Output(scaler, "scaler")

And then I call the solid like this :

min_max_scaler_train = min_max_scaler.alias("min_max_scaler_train")
df_X_train, scaler = min_max_scaler_train(df_X_train)

min_max_scaler_test = min_max_scaler.alias("min_max_scaler_test")
df_X_test, _ = min_max_scaler_test(df_X_test, scaler)

But then Dagit complains about a missing Input definition for the scaler, in the configuration.

Is it possible to have a solid that is only invoked in when exceptions occur in other solids? I'm aware of failure_hook, but don't think it's quite what I want. The exception-handling solid needs to have access to the output of another solid. Sort of like:

@pipeline
def dostuff():
    handle = init_stuff()
    try:
        step_a(handle)
        step_b(handle)
        ...
    except:
        handle_exception(handle)

the small Dagit UI improvements over the past few releases have been such nice quality-of-life improvements 👍 good stuff, y'all!

Is there a comprehensive list of available intermediate storage definitions? I am building a distributed pipeline and the s3 intermediate from

dagster_aws

won't be a valid option in production.

<!channel> Hi all! Our next community meeting is this Tuesday, November 10 at 9 am Pacific Time (UTC-7). Link to Agenda We have three presentations scheduled for the meeting: • @Tobias Macey from Data Engineering Podcast will discuss work he’s done deploying Dagster with Pulumi and SaltStack • @matas from Bestplace will talk about how they built a platform for their data analyst team using Dagster • @daniel from the Dagster core team will give a preview of some new features planned for our 0.10.0 release, including run queueing, a new fault-tolerant scheduler, and launching runs based on external triggers. There will also be a Q&A and discussion section for any questions you have. If you’d like to submit a question in advance, feel free to do so here: Questions Form Please DM me your email or fill out this form to be added to the calendar invite: https://forms.gle/K2dni4iPq48GnUEq6. If you aren’t able to make it, we’ll share a recording afterward.

Rex Ledesma just published a new version: 0.9.18. See https://github.com/dagster-io/dagster/releases/tag/0.9.18 for the release notes.

Hi there, just a minor question - not sure if I missed it somewhere in the documentation. Can I provide one or more pipeline configurations (yaml file) while starting dagit or is there a way to a associate a pipeline with a given yaml file? Basically I want to run the pipeline from dagit while automatically loading a prexisting yaml file. Of course, I could just copy/paste the yaml config but I've thought there must be a more convenient way to automatically link a pipeline with corresponding configs (like from command line

dagster pipeline execute -f pipeline.py -e config.yaml

Hi all, if I'm launching a pipeline via dagit - f file.py, is it possible to have it launch the instance of dagit in a tab automatically like a jupyter notebook?

Hi all, I have a solid that I'm using the

my_solid.alias('some alias')

command with and getting this error:

dagster.core.errors.DagsterUserCodeProcessError: (AttributeError) - AttributeError: module 'repository.utils.solids.my_solid' has no attribute 'alias'

Is the

.alias

function still valid on solids or have things changed for how to do this? Thanks!

Suppose that within a pipeline, there's a composite consisting of a series of solids that share data using a local filesystem (because there's too much data to share over the network). As such, I need to ensure that all solids within that composite execute on the same machine. Every other solid in the pipeline, though, does not have any such constraints, and can be freely distributed via a celery executor or whatever. Is there a way of describing this in dagster today? Or do I need to treat the composite as a separate dag run?

Setting that kind of thing up with celery would be tricky, but not impossible. You'll probably need to make some additions to dagster-celery

Does a solid have any access to the event log for the pipeline run? For example, if I wanted to write a solid that uploaded an event log to a location determined by the output of another solid. Think I could also make do with accessing logged events in a failure_hook, but recall reading somewhere that failure hooks don't have access to the events yet.

You can yield events out to the log

You can't read back from it though, AFAIK

That would have to come as an input 🙂

I was wondering about the Dagster log when you want to log from a supporting module(that is doesn't have the

context

object, and are not solids...also they can be a completely different modules). 1. I tried 

print

 and 

logging

 , but I don't see them on 

dagit

when I run the pipeline. 2. I don't see any of my logs using

<http://context.log.info|context.log.info>('My logs here')

in GCP logs(deployed using k8s), only the dagster logs(eg. Engine events...Started process for pipeline (pid: 17024).) can be seen.

D Hi all! Dagster community meeting is starting in 5 minutes at 9 am Pacific Time (UTC-7). If you haven't registered yet, please check in our Agenda to join our zoom meeting. See you in a little bit!

Thanks for having me on to present! The slides for reference are at https://docs.google.com/presentation/d/1TKL9kem6SDyPr0MADOQIRqwvFgHOF7_gJ9Hqdubiuhs/edit?usp=sharing

I'm happy to answer questions here in Slack 🙂

D Hi all! After the meeting we would like to hear your feedback here: https://forms.gle/c1mFgk8BfVbie9T8A

Thanks everyone for the meeting, excited to see dagster and the community growing! Feel free to DM me if you have any questions about our approach

Hey all! RFC for dropping for Python 2. Speak now or forever hold your peace. 🙂 https://github.com/dagster-io/dagster/discussions/3208