Could someone look at my setup and tell me where to make changes to follow best practices?
My limited understanding is that there are several ways to get Dagster connected with AWS Redshift.
Ingredients Used:
Git, ECS, Dagster, Fargate, and Redshift
• docker build —build-arg REDSHIFT_USER=“${REDSHIFT_USER}” —build-arg REDSHIFT_PASSWORD=“${REDSHIFT_PASSWORD” …
• Dockerfile
ENV REDSHIFT_HOST=
some_uri.region.redshift.amazonaws.com
ENV REDSHIFT_PORT=some_db_port
ENV REDSHIFT_DATABASE=some_db_name
ARG REDSHIFT_USER
ARG REDSHIFT_PASSWORD
ENV REDSHIFT_USER=${REDSHIFT_USER}
ENV REDSHIFT_PASSWORD=${REDSHIFT_PASSWORD}
…
• some_resource.py
@resource
def redshift_creds():
return redshift_resource.configured({
‘host’: {“env”: “REDSHIFT_HOST”},
‘port’: etc…
‘user’: {“env”: “REDSHIFT_USER”},
…
})
• some_file.py
@graph
def the_graph():
some_op.get_stuff_from_redshift()
• repository.py
@repository
def some_func():
return [my_graph.
the_graph.to_job(resource_defs={‘redshift’: redshift_creds()})]
• some_op.py
@op(required_resource_keys={“redshift”})
def get_stuff_from_redshift(context):
query = “SELECT * FROM some_table”
result = context.resources.redshift.execute_query(query, fetch_results=True)
context.log.info(result)
• It works! Yay~ Celebration time!!!