The cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability.

dagster

Hi all I am trying to build a ELT pipeline using various AWS services and dbt. But I want to use dagster as a pure orchestrator for tracking and monitoring and not having any data flowing through my dagster code . Is this good a use case of Dagster ? Any comments?

That's largely how I use it and it works great for me. In my case, I use it to execute tasks, queries, etc. on Snowflake with the occasional loading of data through dagster so that I can FTP a file somewhere or for surfacing metadata in the UI (which is a feature you shouldn't overlook).

same here - using dbt on athena, also with numerous SFTP deliveries that channel via dagster

Thanks for the feedbacks. Looking for more details on how can I separate actual compute using the AWS like Lambda, ECS and EC2 or Fargate from orchestration using dagster ?

Your best bet is to ask in the <#C01U954MEER|dagster-support> channel

Hi <@U04M8B4R7NU>  had a doubt while working with assets without I/O.

This is the example mentioned in the documentation..

Here the both the assets are not returning anything.. just there are dependency between the assets ...

import urllib.request # Addition: new import at the top of `assets.py` import zipfile @asset def stopwords_zip() -&gt; None: urllib.request.urlretrieve( "<https://docs.dagster.io/assets/stopwords.zip|https://docs.dagster.io/assets/stopwords.zip>", "stopwords.zip", ) @asset(deps=[stopwords_zip]) def stopwords_csv() -&gt; None: with zipfile.ZipFile("stopwords.zip", "r") as zip_ref: zip_ref.extractall(".")

I want to understand in the above scenario what the use of using @asset instead of @op...
How do they differ from each other when there no I/O involved in assets.. 

Plus how they differ from normal python function.
Please help me understand.

I’m not the right person to answer you here because I use @asset exclusively. It just seems easier that way. I suggest asking in the <#C01U954MEER|dagster-support> channel