Vlad Efanov

02/14/2022, 9:52 PM
Hi all. I have a config file with paths. I wrote some class to extract the paths from the config file. In every run that starts the sensor, a new instance is created. If I run 10 jobs in parallel, 10 instances of the configuration class are created. What is the best way to extract configuration from config file? config file MSA_orchestrator_config.ini:
script_path = ..\..\scripts\
exe_file_path = ..\..\scripts\mp.exe
new_videos_directory_path = ..\..\video_files\new
in_process_videos_directory_path = ..\..\video_files\in_process
finished_videos_directory_path = ..\..\video_files\finished
Config class
import os
from configparser import ConfigParser

class Singleton(type):
    _instances = {}

    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
        return cls._instances[cls]

class MsaConfig(metaclass=Singleton):
    config_object = ConfigParser()
    default_group_name = "CONSTANTS"
    default_configuration_file_path = f'{os.path.dirname(__file__)}/MSA_orchestrator_config.ini'
    print(f'Constructor config_object is: {config_object}')

    def configuration_extractor(self, group_name=default_group_name,
        if group_name not in self.config_object:
            raise "Default group name is not provided"
        return self.config_object[group_name]
Code in the sensor that implements the class:
msa_config_object = MsaConfig()
constants = msa_config_object.configuration_extractor()


02/14/2022, 10:44 PM
hi @Vlad Efanov! do you mind saying a bit more about your use case here / what you'd like to improve over your current solution? while dagster does offer some utilities for reading run config from files (, it looks like you're trying to load configuration from inside of a sensor, which dagster does not have any specific provisions for. the code inside a sensor is mostly just arbitrary python, so best practices for reading configuration will not be any different in the dagster context.

Vlad Efanov

02/14/2022, 11:58 PM
Hi @owen I have a config file with some configurations like paths. For example, I have a sensor that monitors for new video files in a specific folder. I want to be able to load the path to this folder from my config file. If I could add my configs to dagster.yaml and then somehow read it, it would be better.


02/15/2022, 12:24 AM
hi @Vlad Efanov -- the dagster.yaml file is only meant to be used for dagster-specific configuration, so keeping your other config in a separate config file and loading it inside your sensor code is the suggested pattern here.

Alex Service

02/15/2022, 2:18 PM
I found it easiest to build my python code as a package and then use paths relative to the package
Or, if using a container, copy the configs to a consistent base path