https://dagster.io/ logo
#announcements
Title
# announcements
j

Josh Taylor

02/10/2021, 6:17 AM
How come
postgres_url
in
pg_config
is a str instead of a StringSource? This would allow setting the entire url via environment variable in the yaml, the postgres_db is setup like this:
Copy code
"postgres_db": {
                "username": StringSource,
                "password": StringSource,
                "hostname": StringSource,
                "db_name": StringSource,
                "port": Field(IntSource, is_required=False, default_value=5432),
            },
It looks like this would work, but it's static?
Copy code
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_url: "<postgresql://test:test@{hostname}:5432/test>"
a

alex

02/10/2021, 4:57 PM
want to send an easy PR?
ill just take care of it quick
j

Josh Taylor

02/11/2021, 4:58 AM
just noticed this is already fixed ❤️
c

chrispc

06/08/2021, 4:42 AM
Hi @alex I tried to follow this idea by using env like this:
Copy code
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_url: "postgresql://{DAGSTER_PG_USERNAME}:{DAGSTER_PG_PASSWORD}@{DAGSTER_PG_HOST}:5150/{DAGSTER_PG_DB}"
schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_url: "postgresql://{DAGSTER_PG_USERNAME}:{DAGSTER_PG_PASSWORD}@{DAGSTER_PG_HOST}:5150/{DAGSTER_PG_DB}"
event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_url: "postgresql://{DAGSTER_PG_USERNAME}:{DAGSTER_PG_PASSWORD}@{DAGSTER_PG_HOST}:5150/{DAGSTER_PG_DB}"
but I am getting this error:
WARNING:root:Retrying failed database connection
WARNINGrootRetrying failed database connection WARNINGrootRetrying failed database connection WARNINGrootRetrying failed database connection WARNINGrootRetrying failed database connection WARNINGrootRetrying failed database connection Traceback (most recent call last): File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 3212, in _wrap_pool_connect return fn() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 301, in connect return _ConnectionFairy._checkout(self) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 761, in _checkout fairy = _ConnectionRecord.checkout(pool) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 419, in checkout rec = pool._do_get() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\impl.py", line 259, in _do_get return self._create_connection() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 247, in _create_connection return _ConnectionRecord(self) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 362, in init self.__connect() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 605, in __connect pool.logger.debug("Error on connect(): %s", e) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\util\langhelpers.py", line 70, in exit compat.raise_( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\util\compat.py", line 211, in raise_ raise exception File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 599, in __connect connection = pool._invoke_creator(self) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\create.py", line 578, in connect return dialect.connect(*cargs, **cparams) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\default.py", line 584, in connect return self.dbapi.connect(*cargs, **cparams) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\psycopg2\__init__.py", line 127, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) psycopg2.OperationalError: could not translate host name "{DAGSTER_PG_HOST}" to address: Unknown host The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster_postgres\utils.py", line 116, in retry_pg_connection_fn return fn() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster_postgres\run_storage\run_storage.py", line 53, in <lambda> table_names = retry_pg_connection_fn(lambda: db.inspect(self._engine).get_table_names()) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\inspection.py", line 64, in inspect ret = reg(subject) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\reflection.py", line 182, in _engine_insp return Inspector._construct(Inspector._init_engine, bind) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\reflection.py", line 117, in _construct init(self, bind) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\reflection.py", line 128, in _init_engine engine.connect().close() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 3166, in connect return self._connection_cls(self, close_with_result=close_with_result) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 96, in init else engine.raw_connection() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 3245, in raw_connection return self._wrap_pool_connect(self.pool.connect, _connection) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 3215, in _wrap_pool_connect Connection._handle_dbapi_exception_noconnection( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 2068, in _handle_dbapi_exception_noconnection util.raise_( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\util\compat.py", line 211, in raise_ raise exception File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 3212, in _wrap_pool_connect return fn() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 301, in connect return _ConnectionFairy._checkout(self) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 761, in _checkout fairy = _ConnectionRecord.checkout(pool) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 419, in checkout rec = pool._do_get() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\impl.py", line 259, in _do_get return self._create_connection() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 247, in _create_connection return _ConnectionRecord(self) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 362, in init self.__connect() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 605, in __connect pool.logger.debug("Error on connect(): %s", e) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\util\langhelpers.py", line 70, in exit compat.raise_( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\util\compat.py", line 211, in raise_ raise exception File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\pool\base.py", line 599, in __connect connection = pool._invoke_creator(self) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\create.py", line 578, in connect return dialect.connect(*cargs, **cparams) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\default.py", line 584, in connect return self.dbapi.connect(*cargs, **cparams) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\psycopg2\__init__.py", line 127, in connect conn = _connect(dsn, connection_factory=connection_factory, **kwasync) sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name "{DAGSTER_PG_HOST}" to address: Unknown host
a

alex

06/08/2021, 3:46 PM
Copy code
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
        username: 
            env: DAGSTER_PG_USERNAME
        password:
            env: DAGSTER_PG_PASSWORD
        hostname:
            env: DAGSTER_PG_HOST
        port: 5150
        db_name:
            env: DAGSTER_PG_DB
c

chrispc

06/08/2021, 3:49 PM
Hi @alex thank you. I tried that but I am still getting error.
a

alex

06/08/2021, 3:51 PM
that was fast, same error? Did you restart the process?
c

chrispc

06/08/2021, 3:51 PM
dagster.core.errors.DagsterInstanceMigrationRequired: Instance is out of date and must be migrated (Postgres run storage requires migration). Database is at revision None, head is 7cba9eeaaf1d. Please run
dagster instance migrate
.
Original exception: Traceback (most recent call last):   _File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1706, in _execute_context_     cursor, statement, parameters, context   _File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 691, in do_execute_     cursor.execute(statement, parameters) psycopg2.errors.InvalidSchemaName: no schema has been selected to create in _LINE 2: CREATE TABLE secondary_indexes (_
a

alex

06/08/2021, 3:51 PM
did you do what the error says to do yet?
c

chrispc

06/08/2021, 3:56 PM
yes, I run dagster instance migrate but I got : $DAGSTER_HOME: C:\Users\s4957336\Documents\caps Traceback (most recent call last): File "C:\Users\s4957336\Anaconda3\envs\borrar\Scripts\dagster-script.py", line 10, in <module> sys.exit(main()) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\cli\__init__.py", line 45, in main cli(auto_envvar_prefix=ENV_PREFIX) # pylint:disable=E1123 File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\click\core.py", line 829, in call return self.main(*args, **kwargs) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\click\core.py", line 782, in main rv = self.invoke(ctx) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\click\core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\click\core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\click\core.py", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\click\core.py", line 610, in invoke return callback(*args, **kwargs) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\cli\instance.py", line 46, in migrate_command with DagsterInstance.get() as instance: File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\instance\__init__.py", line 344, in get return DagsterInstance.from_config(dagster_home_path) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\instance\__init__.py", line 359, in from_config return DagsterInstance.from_ref(instance_ref) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\instance\__init__.py", line 376, in from_ref run_storage=instance_ref.run_storage, File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\instance\ref.py", line 235, in run_storage return self.run_storage_data.rehydrate() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\serdes\config_class.py", line 85, in rehydrate return klass.from_config_value(self, result.value) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster_postgres\run_storage\run_storage.py", line 88, in from_config_value return PostgresRunStorage( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster_postgres\run_storage\run_storage.py", line 62, in init stamp_alembic_rev(pg_alembic_config(file), conn) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 131, in exit self.gen.throw(type, value, traceback) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster_postgres\utils.py", line 166, in create_pg_connection yield conn File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\contextlib.py", line 131, in exit self.gen.throw(type, value, traceback) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\storage\sql.py", line 77, in handle_schema_errors raise DagsterInstanceMigrationRequired( dagster.core.errors.DagsterInstanceMigrationRequired: Instance is out of date and must be migrated (Postgres run storage requires migration). Database is at revision None, head is 7cba9 eeaaf1d. Please run
dagster instance migrate
. Original exception: Traceback (most recent call last): File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 1770, in _execute_context self.dialect.do_execute( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\default.py", line 717, in do_execute cursor.execute(statement, parameters) psycopg2.errors.InvalidSchemaName: no schema has been selected to create in LINE 2: CREATE TABLE secondary_indexes ( ^ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster\core\storage\sql.py", line 61, in handle_schema_errors yield File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster_postgres\utils.py", line 166, in create_pg_connection yield conn File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster_postgres\run_storage\run_storage.py", line 59, in init retry_pg_creation_fn(lambda: RunStorageSqlMetadata.create_all(conn)) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster_postgres\utils.py", line 80, in retry_pg_creation_fn return fn() File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\dagster_postgres\run_storage\run_storage.py", line 59, in <lambda> retry_pg_creation_fn(lambda: RunStorageSqlMetadata.create_all(conn)) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\sql\schema.py", line 4744, in create_all bind._run_ddl_visitor( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 2081, in _run_ddl_visitor visitorcallable(self.dialect, self, **kwargs).traverse_single(element) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\sql\visitors.py", line 485, in traverse_single return meth(obj, **kw) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\sql\ddl.py", line 846, in visit_metadata self.traverse_single( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\sql\visitors.py", line 485, in traverse_single return meth(obj, **kw) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\sql\ddl.py", line 890, in visit_table self.connection.execute( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 1262, in execute return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS) File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\sql\ddl.py", line 77, in _execute_on_connection return connection._execute_ddl( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 1352, in _execute_ddl ret = self._execute_context( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 1813, in _execute_context self._handle_dbapi_exception( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 1994, in _handle_dbapi_exception util.raise_( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\util\compat.py", line 211, in raise_ raise exception File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\base.py", line 1770, in _execute_context self.dialect.do_execute( File "C:\Users\s4957336\Anaconda3\envs\borrar\lib\site-packages\sqlalchemy\engine\default.py", line 717, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.ProgrammingError: (psycopg2.errors.InvalidSchemaName) no schema has been selected to create in LINE 2: CREATE TABLE secondary_indexes ( ^ [SQL: CREATE TABLE secondary_indexes ( id SERIAL NOT NULL, name VARCHAR(512), create_timestamp TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, migration_completed TIMESTAMP WITHOUT TIME ZONE, PRIMARY KEY (id), UNIQUE (name) ) ] (Background on this error at: http://sqlalche.me/e/14/f405)
@alex I am enthusiastic of dagster. I am trying to convince to my manager to implement this frame in our system.
if I try a string connection , I got the exactly same error even thought I am hardcoding the values: (I already test the connection via SQLAlchemy, it works!)
a

alex

06/08/2021, 4:01 PM
if you google the underlying error
no schema has been selected to create in
it seems to point to some configuration issues on your pg db
c

chrispc

06/09/2021, 12:04 AM
@alex thank you. Apologize for my question. I was thinking that it is related with dagster.
a

alex

06/09/2021, 12:09 AM
No worries, hope things work well moving forward
c

chrispc

06/09/2021, 3:10 AM
@alex actually, my manager asks me about implementing the authentication layer to access dagit and specifics action over specific features (launch pipeline A, access scheduler B, etc). Do you plan implement this as a future step? (I work for financial company, for them the security is very important).
c

chrispc

06/09/2021, 10:16 PM
@alex thanks for your prompt support. I am a enthusiastic developer (I am a bit of beginner) so I would like to start contributing to Dagster (I love open source). Is there a way to work on this feature with a little supervision? especially in this feature.
😀 1
a

alex

06/10/2021, 2:08 PM
You can ask on the issue. I will say, I don’t think this particular feature is beginner friendly given its scope and complexity.
👍 1
3 Views