Hi everyone! At the start of 2020 I was trying to crawl out of the hell that was managing dozens of small-to-medium sized python jobs, scheduled with Windows Task Scheduler (laugh cry) . Today, about 18 months since my first commit on this project, I can say I was able to put a bit of order to this chaos. Dagster played a major role in that and I've learned a lot in the process. This screenshot hopefully will give you a glimpse of the heavy use Dagster has been put through in my organization (running on a t3.medium). Big thank you to everyone at <!subteam^S017AHZGB5Y|@elementl-team> and everyone that has contributed to this project!

Hi everyone, I've read the documentation and played with some test dags but, currently, I struggle with my imagination how an enterprise scale dagster repo looks like. Are there any public repositorys on github where I can see how others managed bigger repos? I'm thinking about a couple hundred Ops and hundred jobs with heavy usage of global parameters and Input/output parameters

After reading https://dagster.slack.com/archives/C01U954MEER/p1643410970528809?thread_ts=1643408948.422459&amp;cid=C01U954MEER and as it is also helpful for myself https://dagster.slack.com/archives/C01U954MEER/p1641556540061000 I created this blog post https://georgheiler.com/2022/02/02/interactive-dagster-debugging/ to outline how to interactively connect to dagster. Perhaps it is useful for others in the community as well (getting started with interacting with the API of dagster).

In case it’s helpful to someone, here’s a quick-and-dirty bash script I created to let me selectively clean the outputs from successful jobs. Let me know if there’s a better approach! When to use: • If you’re working with large, uncompressed data in a dev or local environment and your disk space just isn’t big enough • You want to keep the logs from the runs, just not the IO from each op • You accept that a re-run of the job won’t be able to resume from an intermediate step When not to use: • If you’re uncomfortable with a random guy’s bash script running a recursive find and remove command 🙂 • If you’re in production or otherwise doing anything critically important • Please make sure nothing else lives in

$DAGSTER_HOME

! If you’re unsure, don’t run it How to use it: • Add to your

.bashrc

,

.bash_profile

, or other appropriate file • reload your terminal or

source

the file above •

dagprune name_of_your_op

How it works: The script checks for the existence of a

name_of_your_op.complete

file exists in the logs and if a corresponding IO directory also exists, then removes everything in the run’s directory except for the

compute_logs

directory. Technically, you could specify any op, but the intended use is for cleaning up outputs from successfully-completed runs (we leave failures alone, since we might actually want to re-run them from their point of failure)

dagprune() {
	if [ -z "$DAGSTER_HOME" ]; then
		echo "Make sure to set env var DAGSTER_HOME before running this command"
		exit 1
	fi
	FINAL_OP="$1"
	for rundir in $DAGSTER_HOME/storage/*/; do
		FILE="$rundir/compute_logs/$FINAL_OP.complete"
		FINAL_OP_DIR="$rundir/$FINAL_OP"
		if [ -f "$FILE" ] && [ -d "$FINAL_OP_DIR" ]; then
			echo "Completed run found. Cleaning op outputs in $rundir"
			pushd $rundir > /dev/null
			find -type d -not \( -name 'compute_logs' -or -name '.' \) -exec rm -rf "{}" \;
			popd > /dev/null
		fi
	done
}

I build a Dagster/Azure Key Vault integration I would love to get feedback !! https://github.com/Martin-Carlsson/Dagster-AzureKeyVault And if you like it, remember to star the repo

Hey folks! I just open-sourced my Dagster project boilerplate, available here! I would love any feedback, especially around ways to make it more useful Notable Features That Make Your Life Easier*:* • Unified Dockerfile for development & deployment; easily integrates with CI/CD processes • Picks up code changes immediately (just hit

Reload

in dagit; don’t have to restart the container!) • Tractable package management using

poetry

. No more hideous

pip freeze > requirements.txt

• Packages the source code according to PEP517 & PEP518 In the near future, I also plan on releasing a technical blog post that dives into how poetry works and how the Dockerfile was designed

Together with @sandy I created a blog post about various topics when getting started with dagster: https://georgheiler.com/2022/03/04/modern-data-orchestration-using-dagster/ I hope this is useful for others as well. For me the motivation was to 1) experiment with dagster (to learn and remember) 2) make it easy for others to follow along (especially also in an classical enterprise context where no additional cloud services should be required to follow along with the examples).

Hey everyone! I’m also excited to (finally!) share my technical blog post detailing how my repo above is able to hot-load code changes in a container and how poetry can be used for package management. Any feedback is welcome! • Blog Post • Github repo

Hey everyone! We invite you to fill out the 2022 State of the Workflow Orchestration Survey; tell them how you're using Dagster! 😄 🎉

Hi everyone 👋 What is the best showcase (blog post, piece of code, etc.) that highlights Dagster in building a data warehouse (eg: Data is in Redshift and you create batch transformation jobs) ?

hey all, I’m Lorenzo from overstory.ai 👋 . As FYI we recently started adopting Dagster and we liked it so much that myself and @Andrea Giardini will be giving a talk at PyConIT on it in June! do we win any free stickers?!

Hey y'all - I published an article about deploying Dagster on ECS using GitHub Actions for CI/CD. I'd love to get feedback from the community - What can be better? What would you like to read about in a future post? What's your dog's name? Thanks in advance for reading 🙂 https://medium.com/@kvmh/lyn-healths-data-laboratory-43806ca4051

Hi everyone! I'm an ML Engineer working to build and ship recommender systems using Dagster. Love the tool, great job devs! While learning Dagster, I found myself hunting for a start-to-finish working example - a Minimum Viable Pipeline. I couldn’t find one, so I built it myself. I wrote it up in a blog post here, or you can directly clone and run the Github repo. This simple example just processes a string (select the input, change case, strip punctuation), which I hope can be an abstraction for all kinds of complex Data/MLOps. It's just 120 lines of code, and explains all the Dagster basics you need to get something running. Thanks for reading!

gave dagster cloud a shout-out in our latest whatnot eng blog post https://medium.com/whatnot-engineering/tuning-whatnots-data-platform-for-speed-and-scale-28d2b5993b42

I made a little SSMSource config schema Field Shape that I've found super useful as I store a lot of parameters in AWS Systems Manager Parameter store that get updated by other CI/CD processes but need to get pulled in to run configs at runtime. I thought other people might find it useful so I put it in a gist to share: https://gist.github.com/zyd14/1280e424e32d81e5bfb74eb81913706a it basically allows you to do this:

@op(
    config_schema = {'param': Field(SSMSource)}
)
def do_something(context):
    ssm_param_value = context.op_config['param']
    ...

ops:
  do_something:
    param:
      ssm_config:
        param_name: some_param_in_ssm
        encrypted: True
OR
ops:
  do_something:
    param: some_scalar_value

and then some_param_in_ssm gets replaced with the value found in AWS Systems Manager Parameter Store at runtime. my only hangup with it currently is that it needs a boto3 connection outside of when you have access to dagster context resources, so it's a bit harder to mock that part out if you're testing run configs that try to retrieve values from SSM - but generally you can pass scalar strings into you run_config when testing instead of using the ssm_config selector to avoid the boto3 connection.

I use dagster in my day to day job, but did you know you can also abuse dagster for day to day issues? In this blogpost I'm reading planning information from a google sheet and write the results to google calendar. I'm running this on a low powered raspberry pi 3b. If you want to see more, look at the github repo Of course running dagster like this misses all the great parts of dagster: • It is not resilient at all, dagster, deamon and run storage all run on the same machine • Version control and modern CI/CD are all but non-existent • I can't use pandera because I can't install arrow on this device • This thing is really really low specced. quadcore 1.4Ghz ARM 1GB RAM vs my laptop: 16gb RAM i7 2.2GHz (also 4 cores)

I wrote about the trends in data orchestrations, and of course, dagster is a big part of it 🙂 Hope it's useful. I'm covering topics such as: • The shift from Data Pipelines to Data Products • Data-aware Pipelines that know about the inner life of a task • Why a declarative approach with higher-level abstractions helps • How abstractions improve reuse code between complex cloud environments • Where do we come from with the evolution of Data Pipeline Orchestration • Open-source data orchestration tools such as Airflow, Prefect, Dagster https://airbyte.com/blog/data-orchestration-trends

👋 Hello, team!

Hey everyone! I wrote a post about how we automated the deployment of (new) user code repositories. It enabled us to migrate very quickly and enabled more and quicker innovation within our company. Thanks for that Dagster daggy love I’m making the point that the deployment of system and user code are not strictly separated when it comes to new repositories. I also have some suggestions on how to improve this and I’m happy to elaborate with you on this.

Hi everyone, I'm Claire from the Dagster team 👋 Wanted to share a post I wrote with @sandy describing how you can use Dagster to lay foundations for future data products while you're building your current one: Roman roads in data engineering: stop starting from scratch whenever you write a data pipeline

Great post from Thinking Machines and the tweet to go with it. 😄

We're excited about the new Hightouch Dagster integration. Read more about it here! Tweet. 🎉 🌈 daggy love

Hi guys i've creating myself project about elt processing use dagster, clickhouse and dbt. If you have any comments please let me know. https://github.com/kotekaman/elt-with-dagster

Hi all, recently I started exploring Dagstermill. It felt like the docs were a bit limited on how to pass output from 1 notebook to the next notebook. I was able to figure it out and wrote an article about it. For reference, see article below, it includes a link to a repo that contains a code example. https://medium.com/@stefan-samba/dagstermill-passing-notebook-results-downstream-in-3-steps-58c22d360f89

My team and I developed a simulation platform using Dagster to test out new machine learning models and logistics optimization approaches. In the process, we learned a ton about our marketplace, integrating batch and production systems (Dagster was key here), and how to accelerate innovation in the tech workplace. Hope you enjoy reading our new article with details of both why and how we did it and let us know your feedback and experiences in simulation. https://doordash.engineering/2022/08/16/4-essential-steps-for-building-a-simulator/

https://dagster.io/blog/evolution-iq-case-study - the case study is back 🙂 enjoy!

so I just ran a historical sync of all the data we use from one of our systems. helluva graph: kudos to the cloud team for making this view so much faster than it used to be

It’s Wednesday and time for a substack post! This time going back into orchestrators with a first look at Dagster Cloud. The first part of a series of posts. Enjoy!

In case you missed it - @Tamas Foldi’s recent analysis of Dagster + Airbyte + dbt: https://medium.com/starschema-blog/dagster-airbyte-dbt-how-software-defigned-assets-change-the-way-we-orchestrate-ac70bb29d640

Hi all! Such a question: I want to assign a launch priority for ops within the same job. How can i do this?