https://dagster.io/ logo
Join Slack
Powered by
Does Dagster have any utilities for merging run da...
# ask-community
c
Does Dagster have any utilities for merging run databases? I've got two separate history/storage folders and i'd like to merge them into one pool of runs if possible. Any tips?
🤖 1
t
What storage backend are you using? The logic for how it'd work would be dependent on that. As far as I know, we don't have utilities to merge databases together though.
c
sqlite backend
What exactly does dagster rely on for populating the 
runs
page? If i just merge the 
runs
table for runs.db, and the 
event_logs
table within index.db, do you think i'll be OK? by merge i mean simple UNION
Looking for some more help here. I've gotten pretty far merging the two tables that I mentioned above. However, i think im still missing some data, because runs that i've imported via the merge still say: "Unable to build execution plan" when loading their run page. Are there any docs about the various tables available within the sqlite storage backend? And what data is required to reconstruct an execution plan?
at the very least a schema of the expected data when loading a run details page would be helpful
Is there a way to view the dagit logs? running 
docker logs
on my dagster-dev container doesn't actually show me errors from Dagit. Specifically looking to find the error message that indicates why my UI is 
Unable to build execution plan
t
Hey there. thanks for asking these questions. I've triaged this over to the team to help out.
c
Thank you!
hey @Tim Castillo, any idea if there's a timeline on when I might be able to get some insight from the team here? any docs or schemas would be a good enough starting point, or even just a pointer to the source code that utilizes the schema of these databases
t
Hi. Just followed up with the team
🙏 1
Alright, just talked to the team. No specific guidance or plan for utilities on this, but the recommendation would be to merge all tables and not specific ones
c
is there a location in the Dagit code that I can look at to understand this further? A pointer directly to github would be super helpful. The guidance of "merge all tables" doesn't help very much given that merging is dependent on what kind of data is in each table. I was able to infer what the tables should look like for 
runs
, 
event_logs
, 
asset_keys
, but for other tables like 
secondary_indexes
for example, I have no idea how I should be merging those, if at all. Totally understandable that y'all don't plan on making any utilities here, but I could really use a pointer to the code so that I can dig in on my own. Unfortunately, having never looked at the codebase, i'm not sure where to even start tracing from to figure out how dagster dev works specifically in the SQLITE storage backend use-case.
Open in Slack
PreviousNext
Powered by