for those who use multiple loggers to keep overly verbose compute logs out of Dagit (not to be confused with the PyPI package of the same name)

has anyone ever tried running databricks-connect in their user-code to run spark code within dagster ops on a remote databricks cluster?

Hi folks! I have got lot of jobs with different configs. how do you organize them? Specify account names maybe?

@won Have you looked at documentation for workspace.yaml? Also, I make folders for ops, graphs, and repositories. Then group jobs from dagster repository abstraction. Of course, there are many ways to organize your codebase.

d*ster everywhere!

Hi, I’m new to Dagster, and have a couple of simple questions that I can’t seem to figure out by reading the documentation. Let’s say, there is a task (C) dependent on task (A) and (B), where A and B are independent of each other. I could write a Python script where Solid A, B will be run from C, and they all run successfully. 1. How can I instruct Dagster to run A and B concurrently? 2. If I need to re-run A and C, but not B, how do I do that without changing the script?

In case it helps, this is how I can handle the above in Luigi. 1. In Luigi, a task definition includes a list of dependencies, and Luigi automatically runs independent tasks concurrently. 2. In Luigi, I could remove output from Task A and C, and rerun Luigi. That way, Luigi knows not to rerun B.

What is the best way to start porting Airflow DAGs from airflow to dagster? I’ve been trying to use this method without much success https://docs.dagster.io/_apidocs/libraries/dagster-airflow#dagster_airflow.make_dagster_repo_from_airflow_dags_path Getting errors like this:

[2022-03-03 08:37:40,862] {api.py:539} INFO - Started Dagster code server for file data_eng/jobs/make_dagster_pipeline.py in process 87160
/Users/khamlin/.pyenv/versions/3.8.8/lib/python3.8/site-packages/dagster/core/workspace/context.py:556: UserWarning: Error loading repository location make_dagster_pipeline.py:dagster.core.errors.DagsterInvariantViolationError: No jobs, pipelines, graphs, asset collections, or repositories found in "make_dagster_pipeline".

This seems #random enough to be in this channel... I'm working on a Medium article about my team's implementation of Dagster. The plan is to publish it at the end of this month with a press release on our brand site. Our brand/marketing leader asked me if I thought anyone from Elementl would be interested in submitting a quote for the press release (sometimes when we send out a release that has to do with another company, they like to say something) so I thought I would just ask here. Is that something y'all would be interested in?

Hello, I recently upgraded our Dagster deployment from 0.12.x to 0.14.x and am trying to start developing using the new API (

ops

,

jobs

,

graphs

, etc). I noticed that

PresetDefinition

is being deprecated and am just curious as to the reason behind the decision and if there is something that replaces it where we can select pre configured run configurations from Launchpad. Happy to be pointed to an article that touches on this topic.

Hello! Does dagster support versioning of data assets that goes through the pipeline?

Hi all, Just a general question, should

workspace.yaml

and

dagster.yaml

be located in same directory/would any issues arise from having them in different ones?

Helllo! I am Harsh and work as a data scientist. Looking to implement a ML workflow with airbyte and dagster. Any resources / tutorials are appreciated. Thank youuuu!

How do people balance the engineering benefits of backfills (testability / idempotence) against the runtime benefits of having a different flow for an initial load? In my case a huge psycopg2 copy_from to insert into Postgres and MongoDB collection call without any filter (executed in large batches). Or are backfills more often just used when a job's been failing for a few days?

Good question. I guess it depends on the product logic behind the data and your day to day intervention on the pipelines themselves. In my case, I get data from vendors with broken APIs or FTP dumps. Backfills can be useful to rebuild the missings bits from the UI. Likewise, the commercial team asks for a rebuild outside of schedule : you rematerialize only the subset of partitions that you need.

Hello, I was trying to find some info how to retrieve run_id when you run Databricks job from Dagster. It prints nice this information into the console but how can i catch this info by code ? I saw that someone created ticket for this one -> https://github.com/dagster-io/dagster/issues/3495 I've used snippet from documentation https://docs.dagster.io/_apidocs/libraries/dagster-databricks I will appreciate your help :)

Fully did a double take on if Ryan and Nick were related 😆 https://dagster.slack.com/archives/CCCR6P2UR/p1648756010442669

We’re an ML company using dagster, considering Databricks and Vertex AI for our MLOps framework. Anyone with hands-on experience with these that can offer feedback?

Additionally, we’re evaluating analytics/BI frameworks that have native support for DBT. Looker and Tableau do not work well with DBT. Any recommendations?

Hi folks, I’m learning dagster and prefect for MLOps and data engineering. While two frameworks have several advantages over AirFlow such as dynamic flow or python functional API in common, I am not sure about the difference between dagster and prefect. Some people say dagster is data-driven and prefect is task-driven, but I am not sure how does this difference lead to differences of concrete appropriate usecases. Could anyone explain advantages (and disadvantages) of dagster over prefect??

My two cents on this : Prefect provides tooling for running tasks on a schedule in a large variety of setups. It has few abstractions, which make it more like a swiss army knife for orchestration. Dagster on the other hand focusses more on data engineering issues like data integrity, testing, idempotency, etc. It has abstractions for solving data problems such as partitions, backfill, assets and resources (which do not exist as such in Prefect, to my knowledge). Both are great tools, and you can probably achieve the same result with both, but Dagster will probably be more suited for managing complex data pipelines and Prefect is probably more suited for simple, general purpose task management.

Is the dynamic operation similar to general concept of lazy evaluation of functional programming languages? https://docs.dagster.io/_apidocs/dynamic

Hey guys, I am new to using Dagster and deploying stuff. The question might seem really silly, my apologies for that. I followed an online example and able to run a simple dagster pipeline locally using docker. (I used docker compose command in the local terminal and then ran dagit command I got for the container) Now, how do I run this in dev environment using docker?

Has anyone been running into data quality issues with cross-db data replication e.g. Postgres > Snowflake? If so, how did you go about validating the data consistency/correctness and fixing the quality issues? Currently working on an open-source package to validate the correctness of db replication and would love more input. If anyone has thoughts about this and would be open to sharing, that’d be super helpful!

Snowflake Book Free!! 📚 https://www.linkedin.com/posts/daniel-portugal_datafriends-snowflake-dataengineering-activity-6927385658449620992-lM9D

At Restack we created a small video about the story of Dagster for you dagster spin I hope you will like it 😊