catherinewu just published a new version: 0.9.6.

general question: is there any appetite/future plans to be able to have solids defined within .ipynb files? so one .ipynb could contain multiple solids rather than the entire notebook being defined as a single solid?

Hey all, we’re opening up a new place on Github for our community to interact and ask questions. Check it out here: https://github.com/dagster-io/dagster/discussions/ Discussions is a great place to: • Ask questions about Dagster • Request features from our team • Share ideas and projects you are working on We’ve historically used Slack as the place to ask questions, but the new default place to ask questions is on Discussions. A lot of the questions asked here in #general are very helpful to the wider community, but get lost in Slack’s history. We hope Discussions becomes a general knowledge base for Dagster in the near future. This Slack will still be very actively used for making announcements, organizing community meetings, having real time conversations, and providing private help to individuals and teams. We encourage everyone to still use this workspace as needed.

You can take a look at our first two proposals from the team here: • https://github.com/dagster-io/dagster/discussions/2904 • https://github.com/dagster-io/dagster/discussions/2902 We’d love to hear your feedback on these ideas

set the channel topic: https://github.com/dagster-io/dagster/discussions/ for questions please 🙏🏻🙏🏻🙏🏻

Yeah just wanted to call out https://github.com/dagster-io/dagster/discussions/2902 specifically. A bit of a doozy that will affect everyone so looking for feedback. tl;dr merge

@pipeline

and

@composite_solid

into a single abstraction:

@graph

So, I have a rough understanding of how to approach this, but given my repository at github.com/mitodl/ol-data-pipelines, and the edx pipeline defined there, I want to set up a preset definition where I can drop some YAML data on disk with my config management system to populate the secrets for two different instances of that pipeline targeting different source systems. Does someone have a quick code snippet for how to do that? If it's more than 5 minutes to do, then don't bother as I can figure it out and I'm just being lazy.

catherinewu just published a new version: 0.9.7.

Hey folks, I'm new to dagster. I believe dagster can show table or node level data lineage graph. Is it possible to show column level lineage using dagster?

This message was deleted.

Hey folks! Is it possible to version your pipelines? Also, is there a way to integrate with GitHub so that pipeline code gets updated when new commits are pushed upstream?

One thing I just noticed is that, in a deployed instance there's a link in Dagit which displays the full contents of the instance file, which includes the database credentials. Is there any work being done to restrict access to that, or elide secret values?

Hi folks, could someone please guide me how to set Dagster up so that dagit / dagster, RabbitMQ and Celery workers are all running remotely (i.e.: on separate EC2 instances)? I have dagit / dagster running on one instance, RabbitMQ on one and multiple instances for Celery workers (need to be able to add more based on workload scaling) I can't seem to figure out how to provide the correct settings / configuration to dagster Also, will some special setup be required for Celery? What app name will have to be passed to it via the CLI?

yuhan just published a new version: 0.9.8.

Hi all, I’m struggling a little with importing custom modules in my

example_pipeline.py

file (I’m using

import src.load_and_store

in the

example_pipeline.py

file (which is in the same directory as

src

). This works correctly when running the pipeline using the Python API (through

execute_pipeline

in _`__name__ == "__main__"`_ in

example_pipeline.py

) and when running it through

dagit

with a

workspace.yaml

file:

load_from:
  - python_file: example_pipeline.py

However, when I try run

dagster pipeline execute -f example_pipeline.py

(from the same directory) I receive the error

ModuleNotFoundError: No module named 'src'

. I’m sure this is a fairly basic import error, but could someone please provide a solution/best practice for this?

Hi all, I am trying to install dagster and dagit on a GCP VM, run the

dagit -f hello_world.py

on it and it say

Serving on <http://127.0.0.1:3000>

how can I access that localhost:3000 via external IP gcp VM? any other config should I set?

@Rizky Eko Putra you could either forward port 3000 via SSH (

ssh -L localhost:3000:localhost:3000 <GCP IP>

I think, I forget the exact magic commands), or pass

-h 0.0.0.0

to your dagit command (which make it accessible to everyone, if your GCP VM is open)

Hello all! I am new to dagster (but am trying to become sufficiently proficient with it) and we are trying to use it at the company to create a data workflow pipeline infrastructure. We already use MLflow but find these two tools largely complementary. When running my first test pipeline I get an error

Error 1: Missing required field "storage" at the root. Available Fields: "['execution', 'intermediate_storage', 'loggers', 'resources', 'solids', 'storage']".

I haven't seen

storage

option in any docs so far. Could you kindly point out me to the resource on how to set this up. We are launching via Python API rather than CLI so I presume it will be some kind of

@storage

decorator? Thanks in advance!

Hi everyone, my goal is to use dagster for automating data jobs in my company. The problem is when deplyoing it to my server and starting the dagit the web interface becomes available for anyone here. Since not everyone should have access to the Web UI I am asking how to restrict access to that? Also there is no obvious Webserver like nginx or apache present in the dagster project. I do not know what to look for. Does anyone have had similar issues or an answer to that?

Bob Chen just published a new version: 0.9.9.

Hi Team, I need to pass a dictionary throughout the solids of a pipeline to capture certain parameters in each solid... is there a better way of doing it apart from using output definitions to output it from a solid and again passing it as input to the next one?

Hello all - a quick questions regarding presets that I could not fully clarify from docs. What is the difference (or interplay) between

run_config

defined in presets https://docs.dagster.io/tutorial/advanced_pipelines#pipeline-config-presets and

execute_pipeline

? When I specify

intermediate_storage

it works when put in

run_config

within

execute_pipeline

but not when I create

PresetDefinition

object that is then passed to

preset_defs

. Thanks for advice in advance!

Hi everyone. I'm trying to use a repository on a gRPC server following the instructions at https://docs.dagster.io/overview/repositories-workspaces/workspaces . I have one docker image for dagit, and another one for the repository, with workspace.yaml and the repository image entrypoint like on that web page. Dagit sees the repository and lists all my pipelines and schedules, but when I try to execute a pipeline, the dagit UI just keeps spinning. There are no logs from dagit, but the repository logs the following:

Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/impl.py", line 184, in start_run_in_subprocess
    run_event_handler=lambda x: None,
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/impl.py", line 133, in _run_in_subprocess
    EngineEventData.in_process(pid, marker_end="cli_api_subprocess_init"),
  File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 943, in report_engine_event
    check.inst_param(pipeline_run, "pipeline_run", PipelineRun)
  File "/usr/local/lib/python3.7/site-packages/dagster/check/__init__.py", line 183, in inst_param
    obj, ttype, param_name, additional_message=additional_message
  File "/usr/local/lib/python3.7/site-packages/future/utils/__init__.py", line 446, in raise_with_traceback
    raise exc.with_traceback(traceback)
dagster.check.ParameterCheckError: Param "pipeline_run" is not a PipelineRun. Got None which is type <class 'NoneType'>.

I can't find anything about this error message. Has anyone seen this? Any ideas what I could do to fix it?

question about dagstermill with passing results-- i've been following the examples provided but I seem to have run into an issue when passing my results from one dagstermill solid to another. This is the typical error received:

dagster.core.errors.DagsterInvalidDefinitionError: In @pipeline LDA_pipeline, received invalid type <class 'str'> for input "base_path" (passed by keyword) in solid invocation "ETL". Must pass the output from previous solid invocations or inputs to the composition function as inputs when invoking solids during composition.

just trying to create some pipelines with dagstermill attached is my .py which defines my solids -- each of the notebooks does execute:

dagstermill.yield_result(...)

Documentation for setting up repositories as python modules / multi location repositories is not quite clear and I think there is scope for improvements.