https://dagster.io/ logo
#dagster-feedback
Title
# dagster-feedback
d

Dagster Jarred

05/01/2023, 3:00 PM
<!here> your friendly neighbor Dagster PM here, reaching out to find Dagster users to talk to for a couple research topics related to customer feedback we’ve been receiving recently: creating visibility into streaming data sources Given how often streaming and batch data architecture co-exist in organizations, some customers want Dagster to serve more as the control plane for adjacent streaming architecture that is (typically) feeding into downstream batch architecture, and wanting both better representations of it on the asset graph, but also to display more metadata about these assets (the definitions, the statuses, etc). Running dagster in a continuous way to ingest data and rapidly trigger runs Oftentimes, Dagster customers are using Dagster to manage data pipelines that service production applications that are mission critical services, or are powering customer-facing experiences. These customers sometimes mention limitations around how sensors work in terms of throughput, or their ability to run beyond the current 60 sec limit. They also mention Dagster pipeline spin-up latency as a limitation for some of these use cases. Sometimes this overlaps with the streaming data use case, i.e. that upstream events are being fired with some rapidity and require response times that aren’t serviced well by rapid fire schedules or sensors. If you’re interested in these topics, or want to share your thoughts on how you’d like Dagster to support you, DM me, or respond on this thread and I’ll follow up with you
❤️ 2