19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
How come `postgres_url` in `pg_config` is a str in...
# announcementsj
Josh Taylor
02/10/2021, 6:17 AMHow come
postgres_urlpg_config Copy code
"postgres_db": {
"username": StringSource,
"password": StringSource,
"hostname": StringSource,
"db_name": StringSource,
"port": Field(IntSource, is_required=False, default_value=5432),
}, Copy code
run_storage:
module: dagster_postgres.run_storage
class: PostgresRunStorage
config:
postgres_url: "<postgresql://test:test@{hostname}:5432/test>"a
alex
02/10/2021, 4:57 PMwant to send an easy PR?
alex
02/10/2021, 5:32 PMill just take care of it quick
j
Josh Taylor
02/11/2021, 4:58 AMjust noticed this is already fixed ❤️
c
chrispc
06/08/2021, 4:42 AMHi @alex I tried to follow this idea by using env like this:
Copy code
run_storage:
module: dagster_postgres.run_storage
class: PostgresRunStorage
config:
postgres_url: "postgresql://{DAGSTER_PG_USERNAME}:{DAGSTER_PG_PASSWORD}@{DAGSTER_PG_HOST}:5150/{DAGSTER_PG_DB}"
schedule_storage:
module: dagster_postgres.schedule_storage
class: PostgresScheduleStorage
config:
postgres_url: "postgresql://{DAGSTER_PG_USERNAME}:{DAGSTER_PG_PASSWORD}@{DAGSTER_PG_HOST}:5150/{DAGSTER_PG_DB}"
event_log_storage:
module: dagster_postgres.event_log
class: PostgresEventLogStorage
config:
postgres_url: "postgresql://{DAGSTER_PG_USERNAME}:{DAGSTER_PG_PASSWORD}@{DAGSTER_PG_HOST}:5150/{DAGSTER_PG_DB}"WARNING:root:Retrying failed database connectiona
alex
06/08/2021, 3:46 PMheres what the schema looks like https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-postgres/dagster_postgres/utils.py#L26-L40
so you should be able to do:
alex
06/08/2021, 3:48 PM Copy code
run_storage:
module: dagster_postgres.run_storage
class: PostgresRunStorage
config:
postgres_db:
username:
env: DAGSTER_PG_USERNAME
password:
env: DAGSTER_PG_PASSWORD
hostname:
env: DAGSTER_PG_HOST
port: 5150
db_name:
env: DAGSTER_PG_DBc
chrispc
06/08/2021, 3:49 PMHi @alex thank you. I tried that but I am still getting error.
a
alex
06/08/2021, 3:51 PMthat was fast, same error? Did you restart the process?
c
chrispc
06/08/2021, 3:51 PMdagster.core.errors.DagsterInstanceMigrationRequired: Instance is out of date and must be migrated (Postgres run storage requires migration). Database is at revision None, head is 7cba9eeaaf1d. Please run
dagster instance migratea
alex
06/08/2021, 3:51 PMdid you do what the error says to do yet?
c
chrispc
06/08/2021, 3:56 PMyes, I run dagster instance migrate but I got : $DAGSTER_HOME: C:\Users\s4957336\Documents\caps
Traceback (most recent call last):
File "C:\Users\s4957336\Anaconda3\envs\borrar\Scripts\dagster-script.py", line 10, in <module>
sys.exit(main())
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\cli\__init__.py", line 45, in main
cli(auto_envvar_prefix=ENV_PREFIX) # pylint:disable=E1123
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\click\core.py", line 829, in call
return self.main(*args, **kwargs)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\cli\instance.py", line 46, in migrate_command
with DagsterInstance.get() as instance:
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\instance\__init__.py", line 344, in get
return DagsterInstance.from_config(dagster_home_path)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\instance\__init__.py", line 359, in from_config
return DagsterInstance.from_ref(instance_ref)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\instance\__init__.py", line 376, in from_ref
run_storage=instance_ref.run_storage,
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\instance\ref.py", line 235, in run_storage
return self.run_storage_data.rehydrate()
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\serdes\config_class.py", line 85, in rehydrate
return klass.from_config_value(self, result.value)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster_postgres\run_storage\run_storage.py", line 88, in from_config_value
return PostgresRunStorage(
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster_postgres\run_storage\run_storage.py", line 62, in init
stamp_alembic_rev(pg_alembic_config(file), conn)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 131, in exit
self.gen.throw(type, value, traceback)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster_postgres\utils.py", line 166, in create_pg_connection
yield conn
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 131, in exit
self.gen.throw(type, value, traceback)
File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\storage\sql.py", line 77, in handle_schema_errors
raise DagsterInstanceMigrationRequired(
dagster.core.errors.DagsterInstanceMigrationRequired: Instance is out of date and must be migrated (Postgres run storage requires migration). Database is at revision None, head is 7cba9
eeaaf1d. Please run
dagster instance migratechrispc
06/08/2021, 3:57 PM@alex I am enthusiastic of dagster. I am trying to convince to my manager to implement this frame in our system.
chrispc
06/08/2021, 3:59 PMif I try a string connection , I got the exactly same error even thought I am hardcoding the values: (I already test the connection via SQLAlchemy, it works!)
a
alex
06/08/2021, 4:01 PMif you google the underlying error
no schema has been selected to create inalex
06/08/2021, 4:02 PMc
chrispc
06/09/2021, 12:04 AM@alex thank you. Apologize for my question. I was thinking that it is related with dagster.
a
alex
06/09/2021, 12:09 AMNo worries, hope things work well moving forward
c
chrispc
06/09/2021, 3:10 AM@alex actually, my manager asks me about implementing the authentication layer to access dagit and specifics action over specific features (launch pipeline A, access scheduler B, etc). Do you plan implement this as a future step? (I work for financial company, for them the security is very important).
a
alex
06/09/2021, 2:15 PMc
chrispc
06/09/2021, 10:16 PM@alex thanks for your prompt support. I am a enthusiastic developer (I am a bit of beginner) so I would like to start contributing to Dagster (I love open source). Is there a way to work on this feature with a little supervision? especially in this feature.
😀 1
a
alex
06/10/2021, 2:08 PMYou can ask on the issue. I will say, I don’t think this particular feature is beginner friendly given its scope and complexity.
👍 1
3 Views