Hi! We're having some scalability issues in our dagster setup and I would like to have the opinion of more experienced users and developers to figure out the best way forward. As I mentioned earlier, we have few pipelines that don't have that many stages, but many datasets go through them. We have modelled our internal concept of dataset as dagster dynamic partitions. However, these datasets are contained by projects and it would be unfeasible to display in dagit absolutely all datasets from all projects (something in the thousands). The user wouldn't be able to filter easily each partition run and, besides that, a user in our system is generally looking for the state of assets in one specific project and not in many, at least not at the same time. To model projects then, since we currently don't have a layered system of partition subdivisions, I'm thinking of putting each project to a separate code location. The definitions would be exactly the same in each code location, but each of them would relate to a single project. I think this might work, but I haven't tested it yet. However, I am a bit worried whether asset keys are shared across code locations (I believe they are if they talk to the same dagster database). If that is the case, does that mean I would have to tag each asset, partition and job name with our project identifier? Moreso I would like to know if this approach actually makes sense.

How big is the dataset? What is the data source exactly? How often does it have to be loaded? If it's less than 10Gb and it should be loaded daily, it might be easier to create a very simple pipeline loading data from all projects all together in one go. Inside each stage (op?) you may load list of projects + credentials and launch export -> import in parallel. Most analytical DBMS will have no issues importing thousands of files or streams in one command.

the data size is actually quite small. We're not yet leveraging dagster's IO managers, we're basically using it to monitor running kubernetes pods which are opaque to dagster. So I'm basically talking about metadata here

Maybe dagit is not the right place to store / monitor this metadata 👀 Sounds like a task for a classic OLTP database + basic custom GUI with filtering.

we'll eventually migrate things to make better use of dagster's infrastructure. What we're doing right now is migrate the current setup we have to dagster so we can make use of simple features like sensors and the asset dependency tracking and visualization

I think this might work, but I haven't tested it yet. However, I am a bit worried whether asset keys are shared across code locations (I believe they are if they talk to the same dagster database). If that is the case, does that mean I would have to tag each asset, partition and job name with our project identifier?

Asset keys need to be unique within a deployment, so you would need to tag the assets with the project identifier. The job name and partitions wouldn't need to be tagged with the project identifier. I think this is a reasonable approach, if the tagging them with the project identifier works for you.

it works indeed! Thanks for the reply