Hi all. I have a config file with paths. I wrote some class to extract the paths from the config file. In every run that starts the sensor, a new instance is created. If I run 10 jobs in parallel, 10 instances of the configuration class are created. What is the best way to extract configuration from config file? config file MSA_orchestrator_config.ini:

[CONSTANTS]
script_path = ..\..\scripts\mp.py
exe_file_path = ..\..\scripts\mp.exe
new_videos_directory_path = ..\..\video_files\new
in_process_videos_directory_path = ..\..\video_files\in_process
finished_videos_directory_path = ..\..\video_files\finished

Config class

import os
from configparser import ConfigParser


class Singleton(type):
    _instances = {}

    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
        return cls._instances[cls]


class MsaConfig(metaclass=Singleton):
    config_object = ConfigParser()
    default_group_name = "CONSTANTS"
    default_configuration_file_path = f'{os.path.dirname(__file__)}/MSA_orchestrator_config.ini'
    print(f'Constructor config_object is: {config_object}')

    def configuration_extractor(self, group_name=default_group_name,
                                configuration_file_path=default_configuration_file_path):
        self.config_object.read(configuration_file_path)
        if group_name not in self.config_object:
            raise "Default group name is not provided"
        return self.config_object[group_name]

Code in the sensor that implements the class:

msa_config_object = MsaConfig()
constants = msa_config_object.configuration_extractor()

hi @Vlad Efanov! do you mind saying a bit more about your use case here / what you'd like to improve over your current solution? while dagster does offer some utilities for reading run config from files (https://docs.dagster.io/_apidocs/utilities#dagster.config_from_files), it looks like you're trying to load configuration from inside of a sensor, which dagster does not have any specific provisions for. the code inside a sensor is mostly just arbitrary python, so best practices for reading configuration will not be any different in the dagster context.

Hi @owen I have a config file with some configurations like paths. For example, I have a sensor that monitors for new video files in a specific folder. I want to be able to load the path to this folder from my config file. If I could add my configs to dagster.yaml and then somehow read it, it would be better.

hi @Vlad Efanov -- the dagster.yaml file is only meant to be used for dagster-specific configuration, so keeping your other config in a separate config file and loading it inside your sensor code is the suggested pattern here.

I found it easiest to build my python code as a package and then use paths relative to the package