Any reason why a dag execution wont exit when a st...
# announcements
v
Any reason why a dag execution wont exit when a step fails? I'm running with dagster pipeline execute
a
hmm what are you seeing exactly? After a step fails the pipeline should fail and the execution should finish. Do you have other steps that are not dependent? Those may get executed first. Some python APIs include
raise_on_error
which will propagate an exception up
v
I was trying to run some code with boto3, but I simplified the code just to get an simple error
i'm reading a file that does not exists
Copy code
@solid
def ecs_fargate_solid(context):
    with open("aa") as f:
        print(f)
    return ''


@pipeline
def docker_solid_pipeline():
    ecs_fargate_solid()
running with
dagster pipeline execute -f solids.py -n docker_solid_pipeline
it hangs at the first line until I hit ctrl c
a
interesting, thanks for the details
v
and I'm using a local postgres as the storages
Copy code
run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_url: "<postgresql://postgres>:@localhost:5432/dagster"

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_url: "<postgresql://postgres>:@localhost:5432/dagster"
a
huh that thread from your interrupt screenshot is set to daemon so shouldn’t block the process from exiting
one workaround is to use
execute_pipeline
from the python API in the interim - either in a seperate .py file or by adding a
__name__ == '__main__'
section to solids.py
v
then i'd just run it wity python solids.py?
a
ya I will debug more in a bit - this might just be an issue with the dagster CLI
v
i've removed the dagster.yaml file of the directory and it failed the same but it exited
a
awesome - thanks for the additional info so it does seem to be due to something in our code around postgres
which version of python are you using? edit: oop 3.7 from the stack trace
v
3.7.4
a
and just to confirm, it hangs indefinitely? There is a 5.0s timeout on that
select
call in the stack trace
v
denitively more than 5s
a
roger
cc @max
v
[<connection object at 0x7f964c901cd0; dsn: '<postgresql://postgres>:@localhost:5432/dagster', closed: 0>]
this is the listen_on param
maybe the connection gets closed before entergin the loop?
m
this may not be fixed in 0.6.6, but is fixed on master
v
allright thank you max
any idea on when this will be released?
a
possibly later this week - you could install
dagster-postgres-nightly
in the interim
v
will do
thank you guys