https://dagster.io/ logo
#ask-community
Title
# ask-community
s

sarah

09/17/2022, 2:04 AM
Hello, I’m trying to get a better understanding of how to use repos and structure our projects using Dagster. We were previously running an older dagster version (0.12.08), and we used different repositories for completely unrelated projects. For example, we had one repository for processing geospatial imagery, another for processing unrelated data arriving from ground sensors, and another for monitoring system log files (we did reuse some code (solids) between different repos, but pipelines were repo-specific). We ran these pipelines in production and development environments (within the same repo) but designated different resources for each. In other words, resources (e.g. destination tables, storage, secrets, etc) were specific to production/development environments, so a specific pipeline could be run in both production or development modes within the same repo. In more recent Dagster versions, it seems we should have different repos for production and development, and then create production and development jobs (with resources defined at this stage) from the same graph. In addition, previously we had multiple sensors to trigger the same pipeline within a single repo (we wanted multiple methods to trigger a pipeline run), but that no longer seems possible and we get the following error:
Copy code
sensor 'x' targets job 'y', but a different job with the same name was provided. Disambiguate between these by providing a separate name to one of them.
We’ve gotten around this by creating additional repos and defining different sensors within each, but I’d like to understand the recommended approach to this type of situation (or maybe I completely misunderstand how repos should be used). Is there some insight/resources that could help me better understand how to structure our projects?
2 Views