What would be the best way to secure a dagit instance set up using dagster-aws?

is there a way to make a python class decorated with

@usable_as_dagster_type

have

required_resource_keys

?

catherinewu just published a new version: 0.7.2.

Hi guys, I've got another question regarding dagit UI. I've got a lot of times the same kind of solid that executes (take a df in input, filter it based on some data, add columns, then output that filtered enhanced df). I have smth like 35 of those and counting, I concat all these filtered df dfs in the end in order to send to the data owners to tell them what is wrong with their data. The thing is all those solids are on the same horizontal line (which makes total sense), but that starts to stretch it a bit. Is there a way to override or arrange the position of these in dagit? If not, no problem, I can live with it 🙂 Thanks!

Stupid question to start with 😅: How you normally develop with spark on a Kubernetes environment? We have Open-Shift with Kubernetes (spark, jupyter notebooks, kafka etc.) and we have a databricks cluster (another spark for fast ramp-up-time). Would you deploy a dagster kubernets pod (https://dagster.readthedocs.io/en/latest/sections/deploying/k8s.html) right? But how would you develop if you're not into Vim? Because I'd like to VS Code locally, but all our stuff is in Kubernetes. What is best practice? I started now to use dagster in a databricks notebook (just to have the concept with solids and reusable pattern). But of course I cannot really visualise the dag etc. and also programming inside a notebook is not really fun.... any hints are well appreciated 🙈

Hey all! We published a medium article on the features and motivations behind the 0.7.0 release! Check it out! https://medium.com/dagster-io/dagster-0-7-0-waiting-to-exhale-b67c2b196982

This message was deleted.

solids and other constructs in dagster are mostly functions, can solid be classes that extends dagster's constructs?

As metadata gets the key of every data project and especially pipelines, I was wondering, if there is a way to have a lineage of a column or and object? Let’s say at the end of your pipeline you write a table, and you would like to know what is the source table for e specific column, what transformation has been made on it. As dagster shows this information already visually, I was thinking, how would you extract this information from code? Do you already have something in mind, or is it already possible? That’s also one advantage from dagster, that all graphs, connections and transformation are in one “tool”. But then it would also be helpful to use these metadata for further essential purposes (e.g. creating or sourcing a data meta catalog). I’m curious to hear your thoughts on this. Thank you!

I am brand new to dagster and thinking of how I can use dagster in my current setting. Basically we are loading a lot of tables from a relational database to another relational database. What I want to do is: Define a solid that accepts databaseconnection and table_name. Input a list of tables to load and make dagster loop that list, inputting table_name in dagster and load in parallell. I have read about Reusable solids in the docs, but have not fully got how configuration can be used. Does anyone have a similar setup?

wondering if there any plans to integrate Dagit with a Notebook front-end like e.g. JupyterLab? at Twitter, most of our user’s laptops don’t have access to interact with the services they need to experiment with and iterate on DS/ML pipelines - so we tend to do most Pipeline/DAG iteration in batch Compute jobs or more interactively in online Jupyter Notebooks (accessed w/ JupyterLab + IDE remoting). it looks like Dagit is already a webapp, so getting it running as a plugin on the notebook server (a jupyter “serverextension”) + an iframe-style JupyterLab extension could be trivial. this should easily extrapolate to other customers with common modes of Notebooks-as-a-DS/ML-Development-Environment (e.g. Google Cloud’s AI Platform) for drop-in Dagster dev support. here’s an example of Dask’s JupyterLab integration for inspiration:

hey guys, this is a bit of a repeat but am going through creating config for solids to wrap some of the pandas functions (

read_excel

and

read_csv

) . For example, being able to define options for

read_csv

in a yaml file like this is great. Having the intellisense in Dagit with config makes it a really useful tool for beginners and others on the team developing pipelines:

solids:
    employees_csv:
      config:
        csv:
          header=true,
          date_format='%m/%d/%Y',
          sep='|',
          ...

However, this is something I keep running into repeatedly. One of the arguments to

read_csv

is

converters

which is defined like this:

converters: dict, optional
Dict of functions for converting values in certain columns. Keys can either be integers or column labels.

How would you represent an argument like this in a yaml file ? Does it even belong in a yaml file despite it being a kwarg like the rest of the args ? If it doesn't belong in a yaml config file, isn't it strange that some arguments are able to fit neatly in the config while others aren't ? I understand these are a bit of a loaded question but what I'm getting at is, instead of yaml files is there any reason using python files to define the

environment_dict

should not be used over yaml ? By using python dicts as the config instead of yaml this would allow the creation of any config dictionary, including covering the case for the

converters

argument above. Thoughts ?

We want to work with dask dataframe as a dagaster type . What is the best strategy for running such process in dagster?

Anyone using Dagster for pipelines where each job is a container? That’s my typical use case using Airflow or Luigi for now. I’ve played a bit with using Dagster for controling jobs on ECS Fargate.

Is it possible to run a backfill locally? If so, how do I configure the run launcher in dagster.yaml?

yuhan just published a new version: 0.7.3.

Hi! I've just spent the past few days browsing the docs and looking into dagster and want to say you've all built something really nice and well-designed. Looking forward to spending more time with it and seeing the project grow, and maybe even contribute too, if I can. Cheers. 🍺

I deeply enjoy building pipelines with dagster so far. Wanted to shout out a big thanks to all the team! So far I created a little pipeline that reads directly from #s3 into a spark dataframe and one that creates me a delta.io table in s3. Now I’m building my transformation logic on top on delta where I can use native SQL (UPDATE/MERGE/ALTER) and ACID transaction on top of s3. Ingesting into Druid and the end deploy on kubernetes. All combined with dagster! Fun times ahead. See some pipelines in action attached 🙂

Can dagster be used for spark streaming jobs? We're currently using dagster for our analytics report pipelines, works like a charm, debugging is so easy and straightforward. Will be building a bunch of internal data products with it in the near future, so will update you guys

quick question, how do I run a pipeline test (pytest) with a yaml file for the pipelines config ?

def test_my_dummy_pipeline():
    res = execute_pipeline(
        my_pipeline,
        environment_dict=<yaml file ?>
    )

Something similar to this except for config:

preset_defs=[
    PresetDefinition.from_files(
        'development',
        mode='development',
        environment_files=[
            file_relative_path(__file__, 'environments/dev_database_resources.yaml'),
            file_relative_path(__file__, 'environments/dev_file_system_resources.yaml'),
            file_relative_path(__file__, 'environments/trips.yaml'),
        ],
    ),
],

Help required - I feel like I'm missing something, but am currently following the tutorial and can't seem to find the execution tab outlined here: https://docs.dagster.io/latest/tutorial/dagit_config_editor

Hi 👋 just joined the channel. I'm setting up a dagster project and have been going through the documentation trying to better understand the config options available for dagster.yaml as well as repository.yaml and the relationship between these two files when

dagit

gets executed. Can anyone point me to further documentation or examples for dagster.yaml etc (beyond what is in https://docs.dagster.io/latest/deploying/instance/)? Maybe what the "default" dagster.yaml would look like and specifically options for

dagit_settings

. Thanks!

One more question 🙂: Is there a way, that we can run dagster as a microservice (execution with the Python API) and store the metadata/logs somewhere central? That Dagit could consume them and visualize the run, and even take up pipeline code and run it manually again(?) This would support our use-case where we run our data-pipelines as a step in our over-all (kafka-event-driven) pipeline. But if we would want to visualize errors on the way, or re-start manually, Dagit could be used for that. Is that use-case foreseen and/or makes sense? Or how would you attack such a case? Is that Dagit as a service deployment? Thanks for any hint or comments.

Hi, sorry if this is a naive question—when running

dagit

how do you specify the mode to run for a pipeline / where can I configure this? I'm getting this error running

dagit

:

dagster.check.CheckError: Failure condition: Could not find mode default in pipeline <PIPELINE_NAME>

because I am trying to run my pipeline in a mode called "development" rather than "default" — when I run via

execute_pipeline

I'm able to pass in the mode via

RunConfig

but I can't find the equivalent way to configure in a yaml file for instance (sorry if this is in the docs and I just haven't been able to locate it). Thanks for any help! (I guess I should caveat and ask, is the only way to specify mode via the dagit UI)

Just curious: how do you guys support your development efforts, like financially?

Hi again, I’m still working my way through the documentation and code base. Let’s say I have some data in Postgres (in RDS) and want to export this to S3, to ingest via Redshift. Is this an appropriate use case? Would I use the dagster libraries for this, or should I be coding the export, copy, import steps from scratch?

Just familiarizing myself with dagster - does it fit the programming model to have a portion of a dag be an external training job such as with gcp's ai platform training? https://cloud.google.com/ai-platform/training/docs/overview

Think I’m a little confused around the FileHandle type and I can’t find any documentation on it. I’m using the

file_handle_to_s3

solid, which expects a FileHandle. My previous step is a postgres job that writes a CSV file to disk. How do I give this file handle to the next task? I tried returning the file name, and returning

open(fn)

but neither seem right.