https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
s

Sup

07/12/2023, 7:46 AM
Hi all I am trying to build a ELT pipeline using various AWS services and dbt. But I want to use dagster as a pure orchestrator for tracking and monitoring and not having any data flowing through my dagster code . Is this good a use case of Dagster ? Any comments?
c

clay

07/12/2023, 12:45 PM
That's largely how I use it and it works great for me. In my case, I use it to execute tasks, queries, etc. on Snowflake with the occasional loading of data through dagster so that I can FTP a file somewhere or for surfacing metadata in the UI (which is a feature you shouldn't overlook).
l

Leo Qin

07/12/2023, 1:01 PM
same here - using dbt on athena, also with numerous SFTP deliveries that channel via dagster
s

Sup

07/12/2023, 7:15 PM
Thanks for the feedbacks. Looking for more details on how can I separate actual compute using the AWS like Lambda, ECS and EC2 or Fargate from orchestration using dagster ?
c

clay

07/13/2023, 3:06 PM
Your best bet is to ask in the #dagster-support channel
s

Sup

07/13/2023, 3:07 PM
Okay Thanks 👍
Hi @clay had a doubt while working with assets without I/O. This is the example mentioned in the documentation.. Here the both the assets are not returning anything.. just there are dependency between the assets ... import urllib.request # Addition: new import at the top of
assets.py
import zipfile @asset def stopwords_zip() -> None: urllib.request.urlretrieve( "https://docs.dagster.io/assets/stopwords.zip", "stopwords.zip", ) @asset(deps=[stopwords_zip]) def stopwords_csv() -> None: with zipfile.ZipFile("stopwords.zip", "r") as zip_ref: zip_ref.extractall(".")
I want to understand in the above scenario what the use of using @asset instead of @op... How do they differ from each other when there no I/O involved in assets.. Plus how they differ from normal python function. Please help me understand.
@clay
c

clay

07/31/2023, 2:48 PM
I’m not the right person to answer you here because I use @asset exclusively. It just seems easier that way. I suggest asking in the #dagster-support channel
s

Sup

07/31/2023, 2:59 PM
Okay sure thanks !
13 Views