Hi All, I recently built a POC for my company using Dagster and got a thumbs up for setting it up to cover more data flows. The POC was a bit rushed, so I didn't organise the code very well. I am wondering if there is a best practices guidelines for greenfield projects? I have questions like - • preferred deployment methods (VM, kubernetes, etc.) • support for gitOps style deployment • preferred environments (linux, windows, although I assume linux 🙂 ) • best practices to store SQL, as most of data transformations are done using SQL (Redshift) • preference for specifying config: yaml Vs python API Thanks in advance!

Hello, I am working on POC (for company) - we have several git repos, each repo contains one dockerizable project (with similar or different package dependencies). I understand each repo as one "workflow" with tasks. Does Dagster support this scenario? Also, is it possible to call task or await task completion from different workflow? Also, what are the best practices for code reuse across git repos/dagster workflows and tasks? Many thanks. 🙂

I see lot of folks here who are trying to move away from Airflow to Dagster so was wondering if anyone can point me to some talks or blog posts that can help me understand the advantages of Dagster DAGS over Airflow DAGS ?

set the channel topic: Next community meeting March 9th 2021! Sign up here: https://forms.gle/Znt4fBt8tpf8tVnN9. Agenda here: https://bit.ly/2Pqz3J8. Subscribe to our Youtube channel http://bit.ly/2XMyY3z for past events videos

Hi all, I’d like to set up a dynamic pipeline which branches into multiple solids at runtime and then waits for all/some amount of them to complete until moving onto a next step. I’m using a solid factory to create the dynamic processes, and I see the “Dynamic Graph” example but it says there’s no “collect” option. Has anyone found a hacky workaround to this?

Hi everyone, I am new to Dagster, I started building my pipelines locally and I would like to deploy it as an Azure App Service. Any suggestions or feedback on the best technologies / setup to follow? Thank you in advance!

yuhan just published a new version: 0.10.8. See https://github.com/dagster-io/dagster/releases/tag/0.10.8 for the release notes.

I was wondering if this sort of a set up is possible with

dagster

? The other two boxes represent hosts with like an agent or a “lite” version of dagster. The agent kicks off processes on the host when main dagster scheduler tells it to.

Hi, ... some idea about this?

Operation name: InstanceHealthQuery Message: too many retries for DB connection Path: ["instance","daemonHealth"] Locations: [{"line":52,"column":3}] Stack Trace: File "/usr/local/lib/python3.7/site-packages/graphql/execution/executor.py", line 452, in resolve_or_error return executor.execute(resolve_fn, source, info, **args) File "/usr/local/lib/python3.7/site-packages/graphql/execution/executors/sync.py", line 16, in execute return fn(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/dagster_graphql/schema/instance.py", line 135, in resolve_daemonHealth return GrapheneDaemonHealth(instance=self._instance) File "/usr/local/lib/python3.7/site-packages/dagster_graphql/schema/instance.py", line 78, in init instance, DaemonType.SCHEDULER File "/usr/local/lib/python3.7/site-packages/dagster/daemon/controller.py", line 236, in get_daemon_status heartbeats = instance.get_daemon_heartbeats() File "/usr/local/lib/python3.7/site-packages/dagster/core/instance/__init__.py", line 1418, in get_daemon_heartbeats return self._run_storage.get_daemon_heartbeats() File "/usr/local/lib/python3.7/site-packages/dagster/core/storage/runs/sql_run_storage.py", line 642, in get_daemon_heartbeats with self.connect() as conn: File "/usr/local/lib/python3.7/contextlib.py", line 112, in enter return next(self.gen) File "/usr/local/lib/python3.7/site-packages/dagster_postgres/utils.py", line 139, in create_pg_connection conn = retry_pg_connection_fn(engine.connect) File "/usr/local/lib/python3.7/site-packages/dagster_postgres/utils.py", line 112, in retry_pg_connection_fn raise DagsterPostgresException("too many retries for DB connection") from exc

Hello, I try to run Dagster with Postgres as a backend for storage. I use this

dagster.yaml

file:

run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      username:
        env: POSTGRES_USER
      password:
        env: POSTGRES_PASSWORD
      hostname:
        env: POSTGRES_HOST
      db_name:
        env: POSTGRES_USER
      port: 5432

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      username:
        env: POSTGRES_USER
      password:
        env: POSTGRES_PASSWORD
      hostname:
        env: POSTGRES_HOST
      db_name:
        env: POSTGRES_USER
      port: 5432

scheduler:
  module: dagster.core.scheduler
  class: DagsterDaemonScheduler

schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      username:
        env: POSTGRES_USER
      password:
        env: POSTGRES_PASSWORD
      hostname:
        env: POSTGRES_HOST
      db_name:
        env: POSTGRES_USER
      port: 5432

run_launcher:
  module: dagster.core.launcher
  class: DefaultRunLauncher

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator

compute_logs:
  module: dagster.core.storage.local_compute_log_manager
  class: LocalComputeLogManager
  config:
    base_dir: ${DAGSTER_HOME}/logs

local_artifact_storage:
  module: dagster.core.storage.root
  class: LocalArtifactStorage
  config:
    base_dir: ${DAGSTER_HOME}/artifacts

telemetry:
  enabled: false

When I run

dagit -f hello.py

I get the following error:

alembic.util.exc.CommandError: Path doesn't exist: '/home/flavien/sandbox/dagster-101/.venv/lib/python3.8/site-packages/dagster_postgres/alembic'.  Please use the 'init' command to create a new scripts folder.

Can anyone help ?

alangenfeld just published a new version: 0.10.9. See https://github.com/dagster-io/dagster/releases/tag/0.10.9 for the release notes.

ing

Hey , I am using solid aliases in my pipeline and wondering if there is a way to specify a couple of configurations in yaml , so that they will be common across the aliases. (In addition to the configuring individual aliases). My use case is this: A solid that inserts values into table (CSV into redshift table). multiple aliases pass on different set of CSVs and table names. Now the SQL script to populate the table is stored in a .sql file which will be same across all calls, and I want to pass that as a config to solid

What is the right dagster abstraction if you want to persist a file to cloud storage (ADLS,S3) with a specific path and then use it in a subsequent solid but have it also work when running locally (to file without cloud storage? The file manager doesn't have ability to choose the file name. Is the best approach to have 1 resource name (i.e. adls2) and then using FakeADLS2Resource for local and just be aware that FakeADLS2Resource stores in memory and not the filesystem? or write an alternate version of FakeADLS2Resource that uses files?

@here I added an #architecture channel

Hi, Is there some way to specify a typechecked Dict in the configuration? Something like

dagster.Dict[str, dagster.Field(dagster.Permissive())]

?

Guys, I wanted to have one or more gRPC servers that provide repositories. I understand I need to make the workspace.yaml looks like the one below: load_from: - grpc_server: host: localhost port: 4266 location_name: 'my_grpc_server My question is: how do I dynamically add new gRPC servers without rebooting Dagster instance? Is there a lazy loading function that could return a list of gRPC servers?

Hi, theres a way to get an asset value from last execution of a solid?

Hi, I am new to dagster and are just getting started. When trying to follow along with the https://docs.dagster.io/examples/airline_demo airline tutorial I observe 2 problems: 1)

pip install -e .

is not installing the optional dependencies which are needed to run the tutorial. 2) https://github.com/dagster-io/dagster/tree/master/examples/airline_demo/airline_demo does not contain the

repository.yaml

file and thus executing a call to

dagit

fails. How can I get the tutorial to work? (for (1) I guess I have installed the necessary dependencies manually, but so far am stuck on (2).

Hi team, I have a doubt regarding composite solids. I want to call the same composite solid from within a pipeline twice with different inputs for a specific solid within the composite, how can I do that programatically?

Hi What is the best way to destroy resource (eg close spark session) after pipeline finish or failure?

Hello team! it that possible to configure mode in dagster configurations files?

Hi, I was thinking about using Dagster as a general workflow engine - e.g. for at user signup flows where certain things need to happen in a specific order. What are your thoughts on that? - and will will it scale?

Hi, how can I use dagster to create pipelines at runtime using metadata in a yaml file?

are there type hints available for the dagster library, either from the package itself or via stubs?

Hi folks, I have a potentially silly question that I’m a little confused about. If I have some output that I want to persist to storage (e.g. filesystem, s3, etc), and I’m not concerned with loading that output into another downstream step, should I still use an IO manager? The documentation & deprecation of file-handlers seems to push in this direction, but for simple use-cases it seems like additional complexity (compared to a resource) for limited payoff?

Hey, I'm following the k8s deployment docs and attempting to get it running on a k8s server v1.15.12. I commented out all of the

startupProbe

sections in the values.yaml but the helm install doesn't work. It looks like the

templates/deployment-user.yaml

enforces that field here: https://github.com/dagster-io/dagster/blob/master/helm/dagster/templates/deployment-user.yaml#L112-L141. I got it working by pulling it locally and removing that block, but is there a way to get it running without doing this? Or am I missing something?