https://dagster.io/ logo
#integration-snowflake
Title
# integration-snowflake
c

clay

03/22/2023, 5:01 PM
so.. in this example, I can't see the .sql file with glob.glob or by just trying to read it as a file. If I ssh into the Docker container, I see the sql file in there.... but I guess I'm not registering it as an asset so it is not being put into the container that's launched when I do a run?
j

jamie

03/22/2023, 5:26 PM
hmm interesting. I’m not very well versed in how the container for an individual run is created and what’s included. If you were to run this locally on your computer does the file get found?
c

clay

03/22/2023, 5:27 PM
No, the file is available and visible inside of the code location container but not when the asset is materialized. The logging of glob.glob in the asset shown here returns an empty list
j

jamie

03/22/2023, 5:27 PM
ok and that happens if you run locally (without docker) too?
c

clay

03/22/2023, 5:27 PM
So I'm thinking the .sql file is not copied into the container that is created when the asset is materialized
In vanilla python, I can see the file
j

jamie

03/22/2023, 5:28 PM
ok cool that makes sense, just wanted to confirm before i loop some more folks in
c

clay

03/22/2023, 5:28 PM
I would guess you could test with a txt file or any other file?
It's just a question, I suppose, of "Why can't I see/find this file that is in the same directory when materializing the asset?"
At the end of the day, it's not a huge deal, but it would really help me to declutter my dagster assets file
If it makes a difference, I'm mounting a volume as the code location. The container name for the code location is
analysis
and the image name is
analysis_image
. When materializing the asset with the code inline, as a string, it works just fine. Dagit is launching an instance of the
analysis_image
to materialize the asset:
[DockerRunLauncher] Launching run in a new container 99b2920fa7d82f5cd8751f9a571a217cce4c769db91d78cc255264428c4990fe with image analysis_image
So, I would think that the instance of
analysis_image
also would mount the code location as a volume? 🤷
(and therefore, the .sql file would be available)
So, thanks, Daniel!
Not sure where you comment went. 🙂
2 Views