https://dagster.io/ logo
Title
a

Akira Renbokoji

11/18/2022, 11:45 PM
Could someone look at my setup and tell me where to make changes to follow best practices? My limited understanding is that there are several ways to get Dagster connected with AWS Redshift. Ingredients Used: Git, ECS, Dagster, Fargate, and Redshift • docker build —build-arg REDSHIFT_USER=“${REDSHIFT_USER}” —build-arg REDSHIFT_PASSWORD=“${REDSHIFT_PASSWORD” … • Dockerfile ENV REDSHIFT_HOST=some_uri.region.redshift.amazonaws.com ENV REDSHIFT_PORT=some_db_port ENV REDSHIFT_DATABASE=some_db_name ARG REDSHIFT_USER ARG REDSHIFT_PASSWORD ENV REDSHIFT_USER=${REDSHIFT_USER} ENV REDSHIFT_PASSWORD=${REDSHIFT_PASSWORD} … • some_resource.py @resource def redshift_creds(): return redshift_resource.configured({ ‘host’: {“env”: “REDSHIFT_HOST”}, ‘port’: etc… ‘user’: {“env”: “REDSHIFT_USER”}, … }) • some_file.py @graph def the_graph(): some_op.get_stuff_from_redshift() • repository.py @repository def some_func(): return [my_graph.the_graph.to_job(resource_defs={‘redshift’: redshift_creds()})] • some_op.py @op(required_resource_keys={“redshift”}) def get_stuff_from_redshift(context): query = “SELECT * FROM some_table” result = context.resources.redshift.execute_query(query, fetch_results=True) context.log.info(result) • It works! Yay~ Celebration time!!!