I am trying to use "build_airbyte_assets" when the connection is only moving csv files to the destination. The official example describe a scenario of postgre to postgres and is using the option "destination_tables" How should be use when is only moving csv files?

This message was deleted.

happy (early) dagster day! we’re live on product hunt https://www.producthunt.com/posts/dagster-cloud

Shower thought: someone should adapt the to Dagster and finish it with “now you’re thinking with assets”

I wrote about Data Lakes and Lakehouse, how to build a Data Lake on top of 1. Storage Layer (AWS S3, Azure Blob Storage, Google Cloud Storage) 2. File Format (Apache Parquet, Avro, ORC) and 3. Table Format (Delta Lake, Apache Hudi, and Iceberg). The next thing would be to add an orchestrator such as dagster to combine all the building blocks. I thought maybe it would be interesting for this audience. If you have any questions, please let me know; happy to chat 🙂

Hello all, I had a question about a use case for dagster cloud? This may be too niche a scenario, but I noticed that containers are a deployment option for the agents. Is there any precedent or mechanism for having dagster cloud jobs trigger a stop/start of those containers the agents live on? This would save on a cloud bill for something that continues to accrue spend when the container is running like ACI on azure for instance.

Random, but just upgraded all our pipelines from the old 0.xx to 1.x. Took about a morning, reason I like Dagster is that if we ever need to change off... it's all just Python code and super easy. Dagster just manages all the fiddly bits for us (and we love it). First commit for us was January 2021, been absolutely rock solid since then. We've been keeping up to date, but we still had a few crusty jobs using solids, but was super easy to just change them to be ops. 10/10 would recommend Dagster.

Hi guys, I'm reading the docs since I'd like to propose Dagster as primary orchestrator for data pipelines at work. We have a mixed HPC/Cloud infrastructure where the Cloud partition is based on Open Stack and the HPC partition is managed by the PBS workload manager.. I would host dagit and the code workspaces/repositories on OpenStack VMs but I would like to spawn the jobs on the HPC partition. Has anyone experimented with a similar setup? From what I understood I would need to implement a custom RunLauncher to integrate with PBS.. Am I correct? If the community has something already backed for such use case I'd love to take a look at it..

I would love to hear your (especially the dagster team) perspective on the new "hot" topic, Data Contracts? 🙂 I see many similarities to SW-defined assets; you too?

FREE 2-week SQL COURSE 👈 👀 My company CoRise is running a FREE SQL Crash Course, starting October 24. Let your friends who want to get into data know! (or for yourself for a refresher) hug Sign up here

Hi there, Just to highlight this to the community. I'm not 100% sure of the implications this might have but

dagster

appears to be relying in dependencies that in case rely on

-ffast-math

, where this bug is found. Apologies if already highlighted as could not find the tread. Should this be raised as an issue? https://moyix.blogspot.com/2022/09/someones-been-messing-with-my-subnormals.html

Some light reading for your Tuesday: https://dagster.io/blog/skip-kafka-use-postgres-message-queue

Recently I wrote an article about Dagster, it caused a big discussion in the data communities here in Brazil. The article is in Portuguese, the title translation is "Bye Bye Airflow, Hello Dagster"... https://www.lucasgabrielb.com/tchau-tchau-airflow-ola-dagster/

Hello folks, I just wrote a blog post on the evolution of data orchestration patterns, and wanted to float an idea for a future of data orchestration which is a more event-driven pattern. • The evolution of data orchestration: Part 1 - the past and present • The evolution of data orchestration: Part 2 - the future I’m open to your feedback or thoughts 🙂

Somewhat random, but the folks at Airbyte are doing a survey on MDS trends, and we would love to have lots of Dagster Community representation in those results - you can jump into the survey here: https://airbyte.com/blog/weve-launched-state-of-data-engineering-survey

Has any of you played with Rust already? I think it will play a significant role in data engineering in the future; still early, but some signs (ideal for DE, most loved languages, etc.) are there. I wrote a small post if anyone has an opinion or is curious. Ideas, comments, and statements are most welcome. I guess working with dagster, someone would wrap the rust library into a python package for now. Or do you have any other thoughts about it? 🙂

@sandy and @Pete Hunt have been playing with Duck DB 🙄 https://dagster.io/blog/duckdb-data-lake

PS: If you enjoyed Sandy and Pete’s article above ^ You will find it posted on Hacker News (on the ‘new’ tab ( https://news.ycombinator.com/newest ) - feel free to upvote it. Thanks.

@sandy getting some love on https://data-folks.masto.host/ ❤️

It’s Wednesday and time for a substack post. The time about how data folks who were on Twitter before have an alternative. Enjoy!

Swagster 😂

New Dagster blog post landed: https://dagster.io/blog/productive-software-engineering

Deploying Dagster with Helm Hi all, I’ve been playing with Dagit locally and I’m beginning to feel productive with it. When it comes time to deploy my code I want to be sure it can handle the work given to it. I talked with a coworker about the best way to do this and he recommended K8s instead of ECS because it’s easier to stick with one or the other and K8s might end up being easier to provision+scale. I’ve never used Kubernetes before but I understand the complexity it comes with. We’re a small team here, it would only really be myself hacking away at EKS. What would you all recommend for me to watch/read/code to help understand Kube+AWS? I’m pretty good with infra in general now and I can get by with ECS, but I want to dig deeper. I want to know all I need to know before diving into this: https://docs.dagster.io/deployment/guides/kubernetes/deploying-with-helm

Happy Friday everyone! We made a pair of Dagster Slack themes to help make this space a bit more Dagster-y. You can apply them right from the quick actions below, or go to Preferences > Themes and paste the hex values there. 🎨 daggy love daggy cowboy 👌 Dagster Light Theme: #F5F4F2,#F0F0F0,#163B36,#FFFFFF,#DAD8D6,#231F1B,#4F43DD,#4F43DD,#4F43DD,#FFFFFF Dagster Dark Theme: #000000,#F0F0F0,#4F43DD,#FFFFFF,#363636,#FFFFFF,#A7FFBF,#4F43DD,#4F43DD,#FFFFFF

message has been deleted

this "Against VendorOps" blog post / Twitter thread made me think of the external resource management framework discussed at the community meeting yesterday. https://twitter.com/vboykis/status/1600112894766481408?s=20&t=CDX7YBisadOTF1GJ1cNlkw

chatgpt knows a bit of dagster

Hello everyone, I'm just here to say that we finished moving all our pipelines from our legacy low-code high-maintenance solution to Dagster. It has been a wild ride, but we received all the help we needed from Dagster support, so a huge thanks for that! So far it has kept its promises : productivity has increased, we do not have to hotfix production anymore and jobs scale. All around a great tool!