Is there any way to do a fan-out of solids? Like an output value is an array and the pipeline iterates over it and sets one solid for each? It seems like that kind of map-reduce model isn't really supported since the DAG has be known statically.

are all of your dagster packages in lock step version wise?

Hey everyone, here is the Zoom link for our community meeting in 15 minutes (9am PST): https://us02web.zoom.us/j/88939083535. Meeting agenda.

Here's a post-meeting survey link https://docs.google.com/forms/d/e/1FAIpQLSdIhRS7VF16gmSpMaRolL3k_JT9ZorIah0gfeU2eUx5lmqdsA/viewform?usp=pp_url We'd love to hear your feedback. The video will be shared soon.

Sorry I wasn't able to make it this time 😕

In lieu of the presentation I was going to give today, I'm happy to answer questions anyone may have about how I'm running Dagster in production.

really enjoying these community meetings and looking forward to seeing how other people are using dagster dagstir

Hello, i'm working with an excel file. I read it with pandas, and i want to do some logic each line (two or three solids), do you thing the best aproach is to use a composite solid to run those solids ? Or pipeline to each line ? Thanks in advance

Does anyone know of a good comparison of Dagster, Airflow (+Cloud Composter, Astronomer), Prefect, and Kedro? It's hard to figure out which framework to use esp. with Airflow v2 coming out. I don't need anything super involved, just a DAG (prefer open source) tool in place of a bash driver running a chain of Py and SQL scripts. The jobs are chained such that the exit status of one determines what happens down the line.

Question about the ECS Launcher (which I know is a work in progress) and launchers more generally.

Hello, I'm running into a

ModuleNotFoundError: No module named '<my_module>'

error when I try to run Dagit. I recently upgraded my project to use the

workspace.yaml

structure. I can provide more details but does anyone have an idea what may be causing the issue?

Is there any sneaky/fancy way to embed HTML into some event output?

So like I could show a little thumbnail of an image when its done processing

Does anyone know if it is possible to run multiple partitions within a single scheduled pipeline run? If so, are there are resources or links showing an example of this?

Are there any examples available for using

dagster-dbt

? It looks like the library has changed quite a bit since I used it previously. Tx!

Dagster made it into this a16z article: https://a16z.com/2020/10/15/the-emerging-architectures-for-modern-data-infrastructure/

Is it possible for a

dagster_type_materializer

to receive multiple materialization configs all at once, instead of being called once per config? For example, the following solid config materializes the Dask DataFrame

facets_df

to both JSON files and a database. This config works (when given real values), but I can’t construct an optimal Dask graph because the Dask

dataframe_materializer()

is called for each config separately. I’d ideally like to be called with both configs, and then compute a Dask graph for them together at the same time.

solids:
  extract_info:
    outputs:
      - facets_df:
          to:
            json:
              path: <s3://bucket/key>
      - facets_df:
          to:
            sql:
              name: table
              uri: <postgresql://server:5432/postgres>

Hey everyone I just posted a discussion that I would love broad participation in. I’m calling it Dagster Python API Festivus: The Airing of Grievances. This is your opportunity to give feedback on our Python API, which we know needs work. We want to know that we are prioritizing stuff correctly or if we have blind spots. https://github.com/dagster-io/dagster/discussions/3113

Also great article from @Michiel Ghyselinck today on using k8s and user/system code separation. https://dev.to/michielhub/dagster-with-user-code-deployments-grpc-2c16

Isaac Hellendag just published a new version: 0.9.15.

Just getting started with dagster and love it so far after doing the first tutorial. The UI is great. Now, hoping someone can point me to a very basic tutorial that shows how to simply load db config and load data to postgresql. Thanks.

Hi guys! I am trying to test a composite_solid and mock a solid within it. I am using monkeypatch mechanism from pytest to replace the solid. However, it does not seem like the patch is working. Do you know how to solve the problem? A sample code is as follows:

def test_my_test(monkeypatch):
    @solid
    def mock_my_solid(_):
        yield Output(pd.DataFrame({
            "date": pd.to_datetime(["2020-05-01", datetime.today()]),
            "name": ["One", "Two"]
        }))

    monkeypatch.setattr("mymodule.my_solid", mock_my_solid)

    result = execute_solid(my_composite_solid_using_my_solid)
    out = result.output_for_solid("my_solid")  # This returns the original output from my_solid

Hey, is it possible to run a pipeline / a specific solid using a rest api / grpc ?

You can use the GraphQL API to launch pipelines

There's an example in the API docs

Hey guys got 2 questions while trying to use dagster to make our python file (which has a lot of if/else conditionals and repeated python functions) more readable and easy to use: • are aliases the only way to reuse a solid multiple times in the same pipeline? • Are conditional outputs the only way to mimic if/else python conditionals?

Hello, maybe there is a visual interface for creating flow by analogy with streamsets?

Helllo, i am currently working with dagster to parse a really long excel file, my startegy involves read a line and execute a pipeline with the data from that line, seems a not so good way to do things. I would like to run it just once, can someone give me a opinion on how to accomplish that. Thanks in advance.