Hey! A couple of (one long / one short) questions:...
# announcements
e
Hey! A couple of (one long / one short) questions: 1. I have two solids in the same pipeline that depend on the same configuration parameter (a folder name). How can I specify a "global" parameter in the configuration of the pipeline run? 2. I have re-created the docker-compose example provided in the docks. However, I have a system that monitors new runs of a specific pipeline and re-starts right where it left off in case it fails. The pipeline writes to a local "checkpoint" file to keep track of its progress. As every run in the docker-compose example is contained in a new docker container, the progress is lost with each run. I thought of setting a volume for the "pipelines" service, but the runs don't seem to share the same volume. Thanks!
d
Hi Eduardo - for the second thing, you don't need to use the DockerRunLauncher in the example if you don't want to. The DefaultRunLauncher (which is used if you leave out the run_launcher field in the dagster.yaml file) will run it within the user code container instead
e
Works like a charm! Thanks @daniel . However, if I wanted to still use the DefaultRunLauncher what would you suggest?
d
you mean the DockerRunLauncher?
e
Yep
d
instead of on the local filesystem, could you write to a checkpoint in a shared database or S3 instsead? then the different containers could all access it
e
Ohhhh, awesome! Will try that out then. Thank you Daniel
s
Hi @Eduardo Santizo - here's how we recommend making a configuration value available to multiple solids in a pipeline: https://github.com/dagster-io/dagster/discussions/3213
e
Thanks @sandy!