Hi there I m exploring Dagster and Dagit this awesome tool t

Question

Hi there I m exploring Dagster and Dagit this awesome tool that you guys gave to the community My question is what s the best way to set the directory `base dir` for sqlite to store run information Can I use the sqlite that comes shipped with Python Currently when it s set to `base dir home stany anaconda3 bin sqlite3` or `base dir home stany anaconda3 envs dagster bin sqlite3` it returns the following error `yaml scanner ScannerError mapping values are not allowed here` Any suggestion Thanks Stany

daniel · Answer

Hi Stany would you mind posting the contents of your dagster yaml file

Stany Angoyi · Answer

Hi Daniel Thanks a lot for willing to help me out Here s the content ` there are two ways to set run storage to SqliteRunStorage` ` this config manually sets the directory `base dir` for Sqlite to store run information in ` `run storage ` `module dagster core storage runs` `class SqliteRunStorage` `config ` `base dir home stany anaconda3 bin sqlite3 home stany anaconda3 envs dagster bin sqlite3` ` and this config grabs the directory from an environment variable` `run storage ` `module dagster core storage runs` `class SqliteRunStorage` `config ` `base dir ` `env SQLITE RUN STORAGE BASE DIR` ` there are two ways to set `event log storage` to SqliteEventLogStorage` ` the first manually sets the directory `base dir` to write event log data to ` `event log storage ` `module dagster core storage event log` `class SqliteEventLogStorage` `config ` `base dir home stany anaconda3 bin sqlite3` ` and the second grabs the directory from an environment variable` `event log storage ` `module dagster core storage event log` `class SqliteEventLogStorage` `config ` `base dir home stany anaconda3 bin sqlite3` `env SQLITE EVENT LOG STORAGE BASE DIR`

Stany Angoyi · Answer

I initially added the sqlite db that I created but didn t work either

daniel · Answer

Ah this is a problem with parsing the yaml file here you can set base dir to env or a string but not both `base dir home stany anaconda3 bin sqlite3` `env SQLITE EVENT LOG STORAGE BASE DIR` The error message isn t very clear here but some YAML parsers include a line number that sometimes gives you a clue where to look to find the problem

daniel · Answer

taking out the home stany anaconda3 bin sqlite3 after base dir should fix it

daniel · Answer

Separate from the specific problem you posted about on my machine at least usr bin sqlite3 points to an executable file that runs sqlite not a directory You probably want to store your runs somewhere specific to dagster rather than in the bin folder The default location if you don t configure the base dir at all is to put the data in a runs subfolder in your DAGSTER HOME folder

Stany Angoyi · Answer

I m using VS Code which is giving me line numbers I did try to store it to home stany runs db but didn t work When I took out the path after base dir I got the following error message `Traceback most recent call last ` `File home stany anaconda3 envs dagster bin dagster daemon line 11 in lt module gt ` `sys exit main ` `File home stany anaconda3 envs dagster lib python3 8 site packages dagster daemon cli init py line 139 in main` `cli obj= pylint disable=E1123` `File home stany anaconda3 envs dagster lib python3 8 site packages click core py line 829 in call ` `return self main args kwargs ` `File home stany anaconda3 envs dagster lib python3 8 site packages click core py line 782 in main` `rv = self invoke ctx ` `File home stany anaconda3 envs dagster lib python3 8 site packages click core py line 1259 in invoke` `return process result sub ctx command invoke sub ctx ` `File home stany anaconda3 envs dagster lib python3 8 site packages click core py line 1066 in invoke` `return ctx invoke self callback ctx params ` `File home stany anaconda3 envs dagster lib python3 8 site packages click core py line 610 in invoke` `return callback args kwargs ` `File home stany anaconda3 envs dagster lib python3 8 site packages dagster daemon cli init py line 30 in run command` `with DagsterInstance get as instance ` `File home stany anaconda3 envs dagster lib python3 8 site packages dagster core instance init py line 292 in get` `return DagsterInstance from config dagster home ` `File home stany anaconda3 envs dagster lib python3 8 site packages dagster core instance init py line 325 in from config` `return DagsterInstance from ref instance ref skip validation checks=skip validation checks ` `File home stany anaconda3 envs dagster lib python3 8 site packages dagster core instance init py line 340 in from ref` `run storage=instance ref run storage ` `File home stany anaconda3 envs dagster lib python3 8 site packages dagster core instance ref py line 220 in run storage` `return self run storage data rehydrate ` `File home stany anaconda3 envs dagster lib python3 8 site packages dagster serdes config class py line 84 in rehydrate` `raise DagsterInvalidConfigError ` `dagster core errors DagsterInvalidConfigError Errors whilst loading configuration for base dir lt dagster config source StringSourceType object at 0x7f0a5771fe80 gt ` `Error 1 Post processing at path root base dir of original value env SQLITE RUN STORAGE BASE DIR failed ` `dagster config errors PostProcessingError You have attempted to fetch the environment variable SQLITE RUN STORAGE BASE DIR which is not set In order for this execution to succeed it must be set in this environment ` `Stack Trace ` `File home stany anaconda3 envs dagster lib python3 8 site packages dagster config post process py line 72 in post process` `new value = context config type post process config value ` `File home stany anaconda3 envs dagster lib python3 8 site packages dagster config source py line 42 in post process` `return str ensure env variable cfg ` `File home stany anaconda3 envs dagster lib python3 8 site packages dagster config source py line 16 in ensure env variable` `raise PostProcessingError `

Stany Angoyi · Answer

The default location doesn t allow me to schedule anything

Stany Angoyi · Answer

Should I create the environment variable SQLITE RUN STORAGE BASE DIR

daniel · Answer

Yeah if you re using the env format in your yaml it assumes that that is an environment variable that you have created and set to the location that you want If you want to specify the base dir yourself that s totally fine and you can set the base dir field directly rather than using env

daniel · Answer

Could you say more about why the default location doesn t let you schedule anything what error does it give you

Stany Angoyi · Answer

Sorry Daniel it was my bad I thought I had to specify the sqlite db that I created for it It s working now slightly smiling face Thank you so much Daniel I now need to go and explore it You guys did an amazing job by developing this awesome tool