still yet to touch scheduling, sensors and jobs but feels really intuitive. good work

If you didn’t get to catch @Pete Hunt’s Blogpost, I recommend checking it out. It’s gone viral. https://dagster.io/blog/chatgpt-langchain

If you get asked about Dagster and want to deflect that call 😆 we now have an updated 10 minute demo / walkthrough you can share:

Not showing any code, but after working with SDAs as the core abstraction for quite a while now, I've observed a subtle but important shift in how I think about my job. I wrote my thoughts on this substack article: https://askvinnie.substack.com/p/now-youre-thinking-with-assets Would love to discuss!

I wrote a post while exploring using Dagster SDAs to build a batch ETL pipeline. https://medium.com/@adebayoadejare/modern-data-platform-orchestration-with-dagster-software-defined-assets-6a7182b0d834

If you have team members who are new to Python and you need them to ramp up fast to work on Dagster, we published Parts 1 and 2 of a new guide (subsequent chapters in the works). We cover the fundamentals of Python projects, packages, pip, dependencies, venv, etc. https://dagster.io/blog/python-packages-primer-1

Our latest blogpost got picked up by Hacker News: Check it out: https://news.ycombinator.com/ The article is

How we deploy faster with warm Docker containers

If you have an interesting Dagster use case or related topic and would like to build your personal brand and/or your organization’s, feel free to reach out to me directly, and I can work with you to write up a case study and/or pitch a talk to a conference, etc. I can make this quite a light lift on your side.

anyone know the zoom demo passcode?

We published a new blogpost this morning: https://dagster.io/blog/fake-stars

That article is quite popular!

By popular demand @Marco Salazar added an RSS feed to our blog: https://dagster.io/rss.xml

Today we published an overview of Partitioning in data pipelines. @sandy is leading the charge on this important work: https://dagster.io/blog/partitioned-data-pipelines

Enjoy this Data Unplugged podcast featuring @Vinnie Watch the video on LinkedIn: https://www.linkedin.com/feed/update/urn:li:activity:7043891508067860480/ …or listen to the podcast: Spotify: https://open.spotify.com/episode/4ztDevPpjRBpbQnDXeoAe2 Apple Podcast: https://podcasts.apple.com/us/podcast/data-unplugged-the-podcast/id1675273913

Hello, for a client I made a small interface to simplify the configuration of notebook dags for people who don't know dagster at all. It's far from an optimal way to use dagster, but they like it, so I thought maybe other people would have use for it. The idea : you declare the notebooks you want to orchestrate/schedule in a toml file and they appeir in dagster. You don't need to know dagster. Who is it for : people who don't know dagster and don't want to learn it. In my case, three data analysts use this for their POCs, before migrating the code into airflow once ready Why is it relevant : not sure 😅 . I prefer to use vanilla dagster but this data analysts (juniors) like it. It can be a first step in the dagster world. https://github.com/nicolasgallandpro/dagyter

Hi, I created my first pipeline using Dagster and I have used assets to build the steps of the ETL process. I am a bit confused about the difference between ops, jobs and assets. My understanding is that Jobs can run Ops and accept configuration (e.g. I can change the behaviour of the code based on execution time). Is that something that cannot be achieved with assets? What about scheduling? Is scheduling more "powerful" when applied to Ops+Jobs vs Assets? I am also considering doing this course: https://corise.com/course/dagster in order to get a more in depth overview of all the functionalities that Dagster offers so that I can build things in the right way from the start. Has anyone tried it? Is it good?

In this edition of @Stephen Bailey's symposium, I present my thinly veiled excuse to shill for dagster's declarative approach (again) https://stkbailey.substack.com/p/nobody-should-write-etl

Hi everyone! We’ve finally converted the Public Utility Data Liberation project (PUDL) to use Dagster! PUDL is an open source project that provides analysis-ready energy system data to climate advocates, researchers, policymakers, and journalists. Dagster has made it much easier to integrate new data into our pipelines and liberate more important energy system data. You can read more about the dagster work in this github discussion. If you’re interested in energy data, climate advocacy, dagster and messy data wrangling we’re always looking for contributors!

New Dagster case study featuring @Zach! https://dagster.io/blog/empirico-case-study

[Note: If anybody else is up for writing up a case study, I am here to help - just DM me if you would like to discuss!]

I've been leisurely working on an integration between dagster and Trino: https://github.com/andreapiso/dagster-trino I think Trino and Dagster are a match made in heaven, because Trino (thanks to its large number of native connectors) can turn all of your data assets into dagster assets with a single integration. No matter whether these assets live on Snowflake, Bigquery, Object Storage/HDFS, MongoDB, etc... This is what the library supports at the moment: • Basic functionalities are essentially 1:1 with dagster-snowflake. The initial cut of this library was basically made by taking dagster-snowflake and re-wire it to use Trino instead. • A

TrinoQuery

type handler that allows the user to pushdown storage and compute of assets to Trino without taking the data out, passing a reference to the Trino table. (the Dagster

DbTypeHandler

system is really amazing, the library heavily use it, super thumbs up for the person who had the idea!) • A set of type handlers that allow to "side-load" Trino data, ie, when using a Trino catalog with Hive metastore, it allows Dagster to automatically find underlying Parquet Data and directly load it from S3/GCS/HDFS/etc... which is a lot faster than using the Trino Client, especially for larger data (i have a benchmark in the example folder showing 10x speed of read even for a small-ish 300MB dataframe). • The type handlers are composable, so a Parquet File type handler is used to build an Arrow Table handler, which is used to build a Pandas handler... it makes it very simple to build custom type handlers. In the example folder i have an example showing a custom Polars type handler in just a couple of lines of code (just converting from/to arrow instead of going all the way to Trino). • I adapted the dagster dbt jaffle shop example to work with dagster-trino instead of duckdb, next i plan to add an example showing the use of ibis (python dataframe library that can be used with Trino) and distributed system such as Spark/Dask/Ray with distributed reads on Trino data using the parquet type handler. The library is still rough around the edges, but if anyone here is interested in having a look and get me some feedback, it would make my day!

Hi All, we recently published a blog how we scaled our metrics computation engine for experimentation using our in house Metrics/Semantic Layer at DoorDash. Dagster was heavily used to orchestrate and manage our computation pipelines. https://doordash.engineering/2023/04/12/using-metrics-layer-to-standardize-and-scale-experimentation-at-doordash/

Materializing Lightning Clustering pipeline ==================================== In my latest post, I perform Exploratory Data Analysis on NOAA's GLM dataset. I create a data pipeline in which I ingest the data and utilize #clustering analysis to create a rough map of lightning clusters . In the EDA #notebook, I load spatial, latitude and longitude raw data and implement k-means clustering algorithm on the test dataset. I evaluate the perform of the 'k' clusters and create a time lapse of lightning clusters. I convert the notebook into a Dagster orchestrated #data pipeline composed of materialized assets performing the required data ingestion and clustering analysis. https://blog.devgenius.io/exploratory-data-analysis-with-lightning-clustering-pipeline-6a2bca17d0d3 https://github.com/BayoAdejare/lightning-map https://github.com/BayoAdejare/lightning-map/blob/master/notebooks/eda_kmeans.ipynb #machinelearning #analytics #orchestration #data #pipeline #dagster #dataengineering #etl

In case you missed it: https://stkbailey.substack.com/p/orchestration-isnt-going-anywhere @schrockn wrapped up the great “Data People Etc.” symposium organized by @Stephen Bailey, sharing his perspectives on data orchestration as a discipline and data orchestrators as tooling.

Hi All! I’ve been using Dagster since the days of solids and pipelines, it has been incredible for our orchestration needs. Really love what the Elementl team has been doing! As a personal project, I’m putting together a repo to quickly spin up a Dagster MVP for your orchestration needs. I plan to extend this repo to include examples demonstrating the many features of Dagster. Please feel free to contribute and let me know your thoughts! https://github.com/sanidhya-singh/dagster-mvp

Hi All! :dagster-spin: We (at Sentra: https://sentra.io/, a data security start-up company from Israel) chose to use Dagster from basically day 1, and we love it. We published a blog including Dagster (as a major component) on AWS’s official containers blog post: https://aws.amazon.com/blogs/containers/how-sentra-manages-data-workflows-using-amaz[…]o-maximize-cost-efficiency-with-minimal-operational-overhead/ While the focus was on AWS’s services themselves (for the AWS containers blog), it was important for us to show Dagster along the way as one of the main key elements in our platform, managing and orchestrating our data pipelines. Thank you all for your amazing product!! 😛artydagster:

I am going to steal somebody’s thunder here, but this is pretty cool: https://airbyte.com/tutorials/implement-ai-data-pipelines-with-langchain-airbyte-and-dagster

Hi everyone, :dagster-spin: I have recently finished working on a Fantasy Premier League (FPL) project using Dagster. I used Dagster to orchestrate a pipeline to extract FPL data for my friends' private league and load it into BigQuery which I then use to service a Dash web app. It was a lot of fun using Dagster and if anyone is interested in the code it can be found here: https://github.com/ajohnson5/fpl-classic-pipeline. I have a lot to learn so would greatly appreciate any feedback or advice for Dagster or data engineering in general! ⚽