https://dagster.io/ logo
Powered by
a

Auster Cid

02/24/2020, 10:43 PM
hi guys, im running into a couple issues with dagster-aws
1. after running dagster-aws init and dagster-aws up, the venv in the created instance is missing dagster-postgres, no biggie, I just pip installed it 2. the dagster.yaml inside dagster_home is missing the hostname and port for my RDS instance 3. after dealing with the two points above it seems I finally managed to start the dagit service, but I still cant access it for some reason
n

nate

02/24/2020, 10:46 PM
hey, yep that sounds like a bug - will take a look
a

Auster Cid

02/24/2020, 10:48 PM
any chance it could be caused by a requirements.txt that includes dagster, dagit and other dagster libraries?
systemctl status dagit says the service is running, are there any logs I can check?
n

nate

02/24/2020, 10:50 PM
if you open a shell on the instance, can you curl dagit?
a

Auster Cid

02/24/2020, 10:53 PM
Copy code
$ curl <http://localhost:3000>
curl: (7) Failed to connect to localhost port 3000: Connection refused
n

nate

02/24/2020, 10:54 PM
hmm ok - let me take a look
a

Auster Cid

02/24/2020, 10:55 PM
hmm, actually it seems the service is not running
yep, just tried it manually
Copy code
Traceback (most recent call last):
  File "/opt/dagster/venv/bin/dagit-cli", line 8, in <module>
    sys.exit(main())
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagit/cli.py", line 152, in main
    cli(auto_envvar_prefix='DAGIT')  # pylint:disable=E1120
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/dagster/venv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagit/cli.py", line 92, in ui
    host_dagit_ui(handle, host, port, storage_fallback, reload_trigger, port_lookup)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagit/cli.py", line 98, in host_dagit_ui
    instance = DagsterInstance.get(storage_fallback)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/instance/__init__.py", line 185, in get
    return DagsterInstance.from_config(_dagster_home())
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/instance/__init__.py", line 207, in from_config
    return DagsterInstance.from_ref(instance_ref)
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/instance/__init__.py", line 215, in from_ref
    run_storage=instance_ref.run_storage,
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/instance/ref.py", line 176, in run_storage
    return self.run_storage_data.rehydrate()
  File "/opt/dagster/venv/lib/python3.6/site-packages/dagster/core/serdes/__init__.py", line 205, in rehydrate
    config_dict,
dagster.core.errors.DagsterInvalidConfigError: Errors whilst loading configuration for <dagster.config.field_utils.Selector object at 0x7f925c651f98>.
    Error 1: Invalid scalar at path root:postgres_db:hostname value "{'hostname': None}" of type "<class 'dict'>" is not valid for expected type "String"
    Error 2: Invalid scalar at path root:postgres_db:port value "{'port': None}" of type "<class 'dict'>" is not valid for expected type "Int"
seems like hostname and post are missing from dagster.yaml again
n

nate

02/24/2020, 11:00 PM
is there a 
dagster.yaml
in 
/opt/dagster/dagster_home
?
it should be in there
but sounds like its not
a

Auster Cid

02/24/2020, 11:01 PM
yes, but it was missing the hostname and port of my RDS instance
I've filled it in and ran the command used to start the service
it's been running for a while, is this normal?
ah just got a traceback, connection to rds timed out
hmm, shouldn't the EC2 and RDS instances be on the same security group?
well, RDS was on the default security group, so I tried moving it to the one dagster-aws created, but it still isnt working
thanks for the quick help @nate, i have to go for today but I'll check back tomorrow
m

max

02/24/2020, 11:20 PM
i think that the generated dagster.yaml may conform to the wrong schema
changed in 9080a545f9c7ed034c1078b05fa70db298acc27f
@nate
a

Auster Cid

02/25/2020, 1:34 PM
heres what I have:
Copy code
/opt/dagster/dagster_home$ cat dagster.yaml
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      username: dagster
      password: <redacted>
      hostname: dagster-rds-root.<redacted>.<http://us-west-1.rds.amazonaws.com|us-west-1.rds.amazonaws.com>
      db_name: dagster
      port: <redacted>

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      username: dagster
      password: <redacted>
      hostname: dagster-rds-root.<redacted>.<http://us-west-1.rds.amazonaws.com|us-west-1.rds.amazonaws.com>
      db_name: dagster
      port: <redacted>
the hostname and port fields had the template placeholders before I fixed it
m

max

02/25/2020, 6:46 PM
ah, i think this is a known issue
oh hm, wait, maybe not
ok, got it
thanks for the report, we will have a fix out in 0.7.2 tomorrow or thursday 🙏
v

vu

02/27/2020, 7:23 AM
Tried it with 0.7.2 and there are still issues with the VPC and security group setup. I eventually got it running. I'm not sure which of the following did it: • Added RDS to the dagster security group • Added in/out rules between both the dagster and default security groups
Powered by