Hello, I’m considering using Dagster for my ETL needs. I deal with image metadata in a digital humanities project. Anyone here would care to chat briefly and help me understand if it makes sense for my use case? I’m afraid it might be overkill and also have a few questions about how ‘atomic’ should solids be, etc. I’d be really cool to talk to someone in the inside or a power user of this tool. Many thanks!

👋🏽 Hello Everyone.

Hi folks, I just came across this example of a simple Dagster implementation and found it really, really helpful. I’ve been struggling a bit to organize my folders/solids/pipelines, as well as configuring a Docker environment for using scheduling and other features, and this has it all in a few, clean, yet real-world files. Hopefully it will help other people getting started with Dagster like myself, and is a great sample of what I’d expect to see in Dagster docs/examples/tutorials

@daniel @sashank Are you guys using event sourcing, for the dagster events? I am just wondering as we are planning to use dagster to run dags for our applications and need a tight integration. And currently I am making a few high level decisions between nodejs/python for our event sourcing mechanism.

🙋‍♂️ Hi all, I'm an engineer working for a UK based insur-tech company. I'm new to this tool but very intrigued by it, I would love if someone familiar with it(or a maintainer/contributor) would volunteer to give our engineering team a quick demo of some use cases 😃 ℹ️

Hi 👋 I am currently evaluating dagster and can't figure out if pipelines can utilize other pipelines / solids from different repositories / environments. Does anybody know or knows if this will be supported anytime in the future? Thanks in advance 🙂

Tfw your pipeline runs successfully the first time 😍 I really really appreciate how nice the dagit UI is

Hi all, We are looking into using Dagster for our workflow management. Of the several requirements, the following are the most important is: • to launch pipelines from Frontend -> I think this can be done from GraphQl API • to create custom DAGs from the frontend -> this can be achieved if the frontend sends YAML or JSON config or through GraphQl API • To be able to orchestrate a DAG from the Solids spread across different containers (which have different programming languages like Python, Java, Julia and interact using REST APIs) If you guys could please point me to docs/resources which could help us in solving these points with Dagster, then it would be helpful to create a strong use-case.

Happy friday all. Has anyone else struggled with test flakiness in data/distributed apps? I’m designing get-flakes to automate triaging flaky tests before they impact velocity. Feel free to send feature requests!

Hi all. I noticed in Dagster, when supplying run-time parameters through the Playground, they are included in the logs. Given these can contian sensitive passwords etc., it could be an idea to include a masking facility for the contents of them, or not to log the playground variable values.

I've been working for a year and a half on a book project with O'Reilly and it's finally published! 97 Things Every Data Engineer Should Know. It's a collection of short essays about different aspects of being a data engineer and building data management systems. It was a lot of fun pulling together the different topics and I hope folks like it. I've got a free trial link for O'Reilly learning that gets you 30 days if you want to read it on there (trial link: https://learning.oreilly.com/get-learning/?code=97TEDESK, book link: https://www.oreilly.com/library/view/97-things-every/9781492062400/). You can also get a physical or ebook copy on Amazon if that's your preference (affiliate link: https://amzn.to/3gOzQit, non-affiliate link: https://www.amazon.com/Things-Every-Data-Engineer-Should/dp/1492062413?crid=VR26MR84HQA&dchild=1&keywords=97+things+every+data+engineer+should+know&qid=1624418681&sprefix=97+things+ever,aps,189&sr=8-3&linkCode=sl1&tag=dataeng-20&linkId=5351fbf75bd8d2d4d8a0fea2ed27b68d&language=en_US&ref_=as_li_ss_tl)

hi all, I’m looking for examples using dagster to commit/pull/push code and data to github, specially using submodules. If anyone has a public repo they’d be willing to share, please send me a link! thank you

Hi All, I am new to Dagster and I want to understand about it workings with SQL Queries. Anyone having a link for end-to-end Demo of a Data Pipeline?

Just wanted to say that I had no idea the name "solid" was controversial after watching the July dagster community video meeting. I was ok with "solid". When I first used dagster, I thought of "solid" from the phrase: "You done me a solid" or "do me a solid". Citation here. So I actually thought that is where the word solid came from 🙂

When is the next Dagster Community Meeting? (I have an recurring invite in my calendar for the second Tuesday of every month which seems incorrect)

hi All we are evaluating dagster for data extraction . have any one tried dagster in production and implement the user signon page to control role based access to dagster ui. appreciate any pointer on this. thank you

I was curious if one can put together a 100% open-source equivalent to what is considered Modern Data Stack and wrote a blog to share my findings. Very curious to hear your thoughts and if I missed any great products!

Hey there, does Dagster support sequencing Zeppelin notebooks in a similar fashion to its Jupyter support?

Random question: Does dagster support webhook events from 3rd party services? Usecase: My goal is to write a solid to call a fivetran connector and then wait until the fivetran sync (to snowflake) is complete. When I look at the airflow-provider-fivetran sensor, it looks like it uses long polling,

poke_interval: Time in seconds that the job should wait in between each tries

, to check the status of the sync job. My goal: fivetran POST webhooks/events to my dagster application, so dagster is told when a sync is done (instead of long polling to see if a sync is done). This obviously adds complexity to the dagster side: I’d need the solid to store state (in a db), that links the solid instance with the fivetran sync job. Then the dagster “application” would need a webserver to receive the webhook, lookup that state, and then complete the corresponding solid. What this sounds like: the ability to add a small Flask application inside of dagster, to handle custom processing (dare I say “serverless” from the perspective of the data engineer). Does anything close to this exist in dagster? Or should I build a custom app which handles all of this and calls the dagster api?

👋 I'm looking into the possibility of using dagster on top of airflow, but it's kind of unclear to me how to achieve good dependency isolation. I see that the

dagster_airflow

module allows for using the Docker operator, and that's great, but I'd like to actually use the

KubernetesOperator

in my situation and it's unclear how to do this.

make_airflow_dag_for_operator

looks promising, but how do I build the

ReconstructableRepository

that it wants? If anybody has experience in this or a good example to follow, I'd be thankful!

anyone uses dagster with aws batch?

maybe data wrangler

For anyone interested in the combination of Meltano and Dagster, I wrote a blog article on how to get started (until Dagster has a plugin for Meltano). If you think I mis-stated something, please let me know. I’m happy to receive feedback and correct errors. https://engineering.widen.com/blog/Dagster-+-Meltano/

Hello all. Is there is common approach for accessing

resource

-like objects from

sensor

currently? I came here from dagster-io/dagster#3794

Dear all, I'm new to Dagster and just managed to put a simple pipeline in place. I'm looking to use it on one of our local VM. It works brilliantly locally. I'm facing some internal pushback in my org regarding the use of Dagster. The main counterargument is we(opposition) will be moving to Azure, and we don't see how this orchestrator complements, is better than azure pipelines or any other orchestrator tool in their ecosystem. I've mentioned dagster ability to translate pipelines as code (my team is very keen on this), simple concepts used as building blocks and ease of deployment. Are there any other points I should refer to? How Dagster complements Azure and areas where it shines deploying python code. Any help is much appreciated. Thank you in advance.

I'm exploring introducing Dagster as an orchestration tool within our DE org. Are there any good references or examples for deploying within Azure?

I'm new to Dagster. I was learning about it when the new release came out. Reading through the new documentation. And I have a few questions It seems to me that the terminology has changed. A Solid is now an Op, a Pipeline is now a Job. Is this accurate?

Hi all. i'v got some trouble with my code. Try to Multi-container Docker deployment Deploying Dagster to Docker with mounting volumes (for user_code) run compose and up 4 containers (

docker_postgresql

,

docker_user_code

,

docker_dagit

,

docker_daemon

) See in UI my jobs. Next go to lachpad and run any of jobs. dagster starting new container with same image as my user_code container. Then i get error:

dagster.core.errors.DagsterInvariantViolationError: Could not find pipeline 'ozon_fbs_job'. Found: 'my_job'.
  File "/usr/local/lib/python3.7/site-packages/dagster/grpc/impl.py", line 75, in core_execute_run
    recon_pipeline.get_definition()
  File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/reconstructable.py", line 112, in get_definition
    defn = self.repository.get_definition().get_pipeline(self.pipeline_name)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/repository.py", line 878, in get_pipeline
    return self._repository_data.get_pipeline(name)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/repository.py", line 658, in get_pipeline
    return self._pipelines.get_definition(pipeline_name)
  File "/usr/local/lib/python3.7/site-packages/dagster/core/definitions/repository.py", line 141, in get_definition
    for found_name in self.get_definition_names()

I haven't got job with

my_job

name 🤯 Where is MY jobs? (

ozon_stocks_job

,

ozon_fbs_job

) 👀 project struct:

dagster_docker/
├───user_code/
│   ├───ops/
│   │   └───ozon_fbs.py
│   │   └───ozon_stocks.py
│   ├───resoutces/
│   │   └───db_resource.py
│   └───repo.py
├───dagster.yaml
├───docker-compose.yml
├───Dockerfile_dagster
├───Dockerfile_user_code
├───requirements.txt
├───workspace.yaml

Wher did i miss?

Hi all, I'm facing an issue regarding concurrency, here is my dagster.yaml and jobs details where i'm setting the tag. But when I run the Dagster all the jobs starts synchronously by sensor instead of only two.

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 10
    tag_concurrency_limits:
      - key: "Reports"
        limit: 2

job details

@job(
  resource_defs={"report_details": report_details.get_details},
  tags={"type": "Reports"},
)

Hi all, I'm facing the issue of slowness like I have a job which takes 3 to 4 minutes to get complete when I run a single job it completes in 3 to 4 minutes but when I run around 32 concurrent jobs the time for each job reach to 1 hour or more even though my ec2 server is large and using one 20% of its resources only, i'm not sure why it's happening that by increasing the concurrent jobs speed getting slow.