At Restack we created a small video about the story of Dagster for you :dagster-spin: I hope you will like it 😊

Don’t forget to register for the BEAM Collage event, totally Free 😄 🐝 https://www.linkedin.com/posts/daniel-portugal_season-2022-activity-6929430204679434242-IZI8

Hi #random, I have written a module which can help in building dagster ops, graphs and repositories in an automated way using a config file. I have used it now in a couple of my deployments and found it helpful. Sharing it here in case you feel it can be used in your project as well. It covers only a few basic features as of now, but I believe you can extend it as per your case. https://github.com/asanoop24/dagger/tree/master

from : Airflow, Prefect, and Dagster: An Inside Look quote : "Dagster takes a first-principles approach to data engineering. It is built with the full development lifecycle in mind, from development, to deployment, to monitoring and observability. Prefect, on the other hand, adheres to a philosophy of negative engineering, built on the assumption that the user knows how to code and makes it as simple as possible to take that code and built it into a distributed pipeline, backed by its scheduling and orchestration engine." newbie comment : really agree with it, I find the seperation of concerns principle is strong in Dagster :baby-yoda: (and I've already learned a lot just by losing myself in the documentation and people's code)

Hey, has anyone ever used Dagster in conjunction to TimeScale DB?

Just read this article and loved it https://www.ethanrosenthal.com/2022/05/10/database-bundling/ (unfortunately they have a go at the unbundling airflow article which I thought was pretty good). In my mind it's basically arguing that the analytics stack has MLOps covered, provided someone comes up with a little more guidance / config in terms of how to build on top of it (like a DBT for Model monitoring instead of Data Warehouses)

Hey folks, we’re looking for a data analytics framework that can be embedded in a web app. Who are the hottest startups out there doing this? We’re evaluating LOOKER and Metabase, looking for more options. Our data lives inside BQ.

👋 Hello, team! here to learn more about scheduling aspects of dagster form the community

Not sure if anyone has run into this, but: I have one postgres instance with multiple dbs (one for my application code, one for Dagster). Been trying to prevent it from ballooning on storage space, and didn't realize that the

event_logs

table can get so very large (it's half my storage). So if you happen to have something eating storage and can't pinpoint it, couldn't hurt to check that table!

Hey folks, we are just getting started with Dagster and we are wondering if we should have two separate deployments of Dagster (dev and prod) or a single deployment where you would configure the jobs differently depending if they're running on dev or prod. Do you have any strong opinions about these approaches?

🧚 :dancing: 🕺 :dancing: 💃 :dancing: 🧚‍♀️ Hello guys, My project's current status I am working on the development of an ML product. So far we have used DVC to define our ML pipeline, including preprocessing steps, training, baselines computations, and so on... We are working with GCP and so far we are running our pipeline using vertexAI training jobs. Where I need your help Now we want to move to another DAG framework that allows us to run the different steps on different machines (with or without GPU) and that is more integrated with

GCP

. I have always considered

kubeflow

as a strong option as it it well integrated with

GCP

and specifically

VertexAI

. Now I saw a talk at Pycon Italy recently about

Dagster

and I felt like it could compete with

Kubeflow

but I still can't figure out whether

Dagster

is better. One specific question I don't have the answer for is if there is an easy way to run the

Dagster

ops on

Cloud Run

or

VertexAI

. Thanks a lot for your help 🙂 your project looks really cool!!! (@Andrea Giardini I saw you at pycon.italy, and contacted you on linkedin, thought that would make sense to ask my question here 🙂 ) 🧚 :dancing: 🕺 :dancing: 💃 :dancing: 🧚‍♀️

Hey I'm new to dagster and was curious if there was any guides on how to write a simple abstraction layer to easily port existing map/reduce code running in MRJob to dagster. My impression is dagster supports a more general computation paradigm than map/reduce and this should be possible, any guidance is greatly appreciated.

Hi all, I work as a database administrator and my python knowledge is very limited. I know control flow and practicing functions now. In order to work with Dagster, which Python concepts should I learn and practice?

Any Dasgter experts here want to do a couple hour consultancy to help us hit the ground running esp on DevOps side...? DM me

I was going to start digging into feature stores, but having just had a peak at Feast's I've realised that the API and functionality can be very similar to Software Defined Assets in Dagster (maybe with a more column centric, rather than table / asset centric view). Would I get much in the way of additional benefits from adopting one?

I wrote a bunch of pipelines to migrate data from Mongo (App DB) to Postgres (temporary budget data warehouse until we upgrade to something like SnowFlake). Realised too late that what I was really doing was defining Airbyte Sources https://docs.airbyte.com/connector-development/tutorials/building-a-python-source and Destinations which are already well defined and robust. Our main Software Engineer is touchy about App DB query load. I'd imagine that Airbyte will be just as efficient as anything I've written if not more so? My mongo queries retrieve all the data in a collection since a timestamp from just before the previous pipeline execution if that makes sense. And then upsert on a key so a tiny bit of record overlap doesn't matter. Basically my pipelines can be a little buggy, should I just migrate? @Stephen Bailey I do use copy_expert for my backfills which makes them speedy, not sure if Airbyte would do that. But I guess the fact that Airbyte takes everything and loads it into json means that I wouldn't need to run a backfill because there's no transformation logic to get wrong or change. Not posting on Airbyte slack because Airbyte devs will obviously tell me to use Airbyte.

Does this matrix make sense for comparison or does it sound too biased ?

I officially launched my new show, The Machine Learning Podcast! The second episode was published earlier this week. For folks who don't know, I also run the Data Engineering Podcast

Hi, I was curious if anyone here has experience with recommendation engines and what a normal amount of memory consumption is? We've built a recommendation engine that provides recommendations but it's consuming a massive amount of memory most likely due reading relevant data into memory via Pandas (read_csv).. We're going to take another look and try to refactor this but wanted to know if this was normal and if i should consider scaling machine instead.. currently we're maxing out a 64 GB RAM machine 😞

We've been having a spirited debate about what we should call the role between Admin and Viewer, currentled called Editor in Dagster. Which do you like best?

hey friends, looking for input from the community on this topic above, if you’re interested in helping us name our roles more effectively 🙂

People keep saying things to this effect, I'm assuming Dagster is in the same camp as Prefect, but I have no idea what this difference in design choice is, and feel like I really should. Edit: I don't have much experience with Airflow. I'm assuming this is an architectural decision (how work is divided on k8s?) and not just ergonomics.

Does anyone happen to have a Pulumi snippet for spinning up AWS EKS + Dagster agent (+ fluentbit) for Dagster Cloud?

I’m teaching a free “Learn All Of Data Engineering In 3 Hours” workshop on Thursday 7/28 at 4:30 ET as part of the Operational Analytics Summer Community Days conference https://www.operationalanalytics.club/summer-community-days

In the docs, for deployment, there is Separate section for "Deploying Dagster to AWS" and "Deploying Dagster to GCP". These section mention in introduction about storing runs and events and IO manager capabilities using the cloud provider. Is there something similar for Azure as well?

[RESOLVED] Our team has been debating whether using f-string formatting is a best practice or not when logging specifically in our repository that uses dagster. There does not seem to be clear consensus in the python community, and the official python docs for python 3.9 (which is what we use) https://docs.python.org/3.9/library/logging.html# doesn’t explicitly say either. This stack overflow post says f-strings has worse optimization, but other people still use it, https://stackoverflow.com/questions/54367975/python-3-7-logging-f-strings-vs. I saw the dagster codebase itself using f-string formatting too. I’d like to switch over to use f-string formatting as well, but want to be able to back up why it won’t cause any issues. Any thoughts?

ayyyyy thanks for the shoutout @schrockn https://podcasts.google.com/feed/aHR0cHM6Ly93d3cuZGF0YWVuZ2luZWVyaW5ncG9kY2FzdC5jb20vZ[…]l=en&ved=2ahUKEwjoj_2qxJn5AhXUM1kFHV6CD7MQieUEegQIAhAI&ep=6

Ahhhhhh… this was nice https://twitter.com/stkbailey/status/1553086006055866369?s=20&t=zwW_KgHTMcssY7G-8x_82w

Hi, Besides the official documentation website where can I learn more about Dagster? good practices, etc. Fast search on google does not give me too much infor