Getting this while trying to add a code location in hybrid m

Question

Getting this while trying to add a code location in hybrid mode using GKE Anyone faced this before

daniel · Answer

Hey Abhishek can you try running kubectl describe job on the job being referenced there spina prod xxx and see if any errors jump out from that description If there s a pod with a similar name in the cluster then running kubectl describe pod on that pod or checking its logs can also explain why its having trouble starting up

daniel · Answer

one common source of errors here is not having the dagster cloud package installed in the image that s running your code but usually there s some clue in the logs there that explains what s going on

Abhishek Agrawal · Answer

thanks a ton Daniel Should I just add the install command for dagster cloud in my requirements file and then re create the docker image

Abhishek Agrawal · Answer

```abhishek agrawal cloudshell ~ mutinex dev $ kubectl describe pod spina prod 1a8182 7b8cc7b94 pmtpr namespace dagster cloud Name spina prod 1a8182 7b8cc7b94 pmtpr Namespace dagster cloud Priority 0 Service Account user cloud dagster cloud agent Node gk3 dagster default pool 12146bc5 c3j4 10 192 0 14 Start Time Wed 22 Feb 2023 03 04 56 +0000 Labels pod template hash=7b8cc7b94 user deployment=spina prod 1a8182 Annotations runtime default Status Running IP 10 107 0 17 IPs IP 10 107 0 17 Controlled By ReplicaSet spina prod 1a8182 7b8cc7b94 Containers dagster Container ID 6 Image Image ID Port lt none gt Host Port lt none gt Args dagster api grpc State Waiting Reason CrashLoopBackOff Last State Terminated Reason Error Exit Code 1 Started Wed 22 Feb 2023 03 08 58 +0000 Finished Wed 22 Feb 2023 03 08 58 +0000 Ready False Restart Count 5 Limits cpu 250m ephemeral storage 1Gi memory 512Mi Requests cpu 250m ephemeral storage 1Gi memory 512Mi Environment Variables from dagster cloud agent token Secret Optional false Environment DAGSTER LOCATION NAME spina DAGSTER INJECT ENV VARS FROM INSTANCE 1 DAGSTER CLI API GRPC LAZY LOAD USER CODE 1 DAGSTER CLI API GRPC HOST 0 0 0 0 DAGSTER INSTANCE REF class InstanceRef compute logs data class ConfigurableClassData class name CloudComputeLogManager config yaml n module name dagster cloud storage compute logs custom instance class data class ConfigurableClassData class name DagsterCloudAgentInstance config yaml dagster cloud api n agent token n env DAGSTER CLOUD AGENT TOKEN n deployment prod nuser code launcher n class K8sUserCodeLauncher n config n dagster home opt dagster dagster home n env secrets n dagster cloud agent token n instance config map user cloud dagster cloud agent instance n namespace dagster cloud n pull policy Always n service account name user cloud dagster cloud agent n module dagster cloud workspace kubernetes n module name dagster cloud instance event storage data class ConfigurableClassData class name GraphQLEventLogStorage config yaml n module name dagster cloud storage event logs local artifact storage data class ConfigurableClassData class name LocalArtifactStorage config yaml base dir opt dagster dagster home n module name dagster core storage root run coordinator data class ConfigurableClassData class name DefaultRunCoordinator config yaml n module name dagster core run coordinator run launcher data class ConfigurableClassData class name DefaultRunLauncher config yaml n module name dagster run storage data class ConfigurableClassData class name GraphQLRunStorage config yaml n module name dagster cloud storage runs schedule storage data class ConfigurableClassData class name GraphQLScheduleStorage config yaml n module name dagster cloud storage schedules scheduler data class ConfigurableClassData class name DagsterDaemonScheduler config yaml n module name dagster core scheduler secrets loader data class ConfigurableClassData class name DagsterCloudSecretsLoader config yaml n module name dagster cloud secrets settings storage data class ConfigurableClassData class name CompositeStorage config yaml event log storage n class name GraphQLEventLogStorage n config yaml n n n module name dagster cloud storage event logs nrun storage n class name GraphQLRunStorage n config yaml n n n module name dagster cloud storage runs nschedule storage n class name GraphQLScheduleStorage n config yaml n n n module name dagster cloud storage schedules n module name dagster core storage legacy storage DAGSTER CLI API GRPC PORT 4000 DAGSTER CURRENT IMAGE DAGSTER CLI API GRPC PACKAGE NAME spina DAGSTER CLOUD DEPLOYMENT NAME prod DAGSTER CLOUD LOCATION NAME spina DAGSTER CLOUD IS BRANCH DEPLOYMENT 0 Mounts var run secrets kubernetes io serviceaccount from kube api access d7m6k ro Conditions Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes kube api access d7m6k Type Projected a volume that contains injected data from multiple sources TokenExpirationSeconds 3607 ConfigMapName kube root ca crt ConfigMapOptional lt nil gt DownwardAPI true QoS Class Guaranteed Node Selectors lt none gt Tolerations op=Exists for 300s op=Exists for 300s Events Type Reason Age From Message Normal Scheduled 6m1s Successfully assigned dagster cloud spina prod 1a8182 7b8cc7b94 pmtpr to gk3 dagster default pool 12146bc5 c3j4 Normal Pulled 5m14s kubelet Successfully pulled image in 44 158385722s Normal Pulled 5m9s kubelet Successfully pulled image in 1 689241565s Normal Pulled 4m51s kubelet Successfully pulled image in 1 717180699s Normal Created 4m22s x4 over 5m14s kubelet Created container dagster Normal Pulled 4m22s kubelet Successfully pulled image in 1 676467814s Normal Started 4m20s x4 over 5m13s kubelet Started container dagster Normal Pulling 3m26s x5 over 5m58s kubelet Pulling image Normal Pulled 3m24s kubelet Successfully pulled image in 1 719730509s Warning BackOff 57s x19 over 5m6s kubelet Back off restarting failed container```

daniel · Answer

yeah pip install dagster cloud alongside dagster Did the logs for the task indicate that it was having trouble importing the dagster clodu module

daniel · Answer

er logs for the pod rather

Abhishek Agrawal · Answer

I could not see it from the describe

Abhishek Agrawal · Answer

ah okay

Abhishek Agrawal · Answer

let me check

daniel · Answer

We re planning to include those pod logs automatically in dagit when this happens instead of that cryptic timeout message which should hopefully make this less confusing in the future

Abhishek Agrawal · Answer

image png

Abhishek Agrawal · Answer

how do I see the logs

Abhishek Agrawal · Answer

sorry I m new to k8s and learning as I go

daniel · Answer

ah that looks different actually one sec

daniel · Answer

How did you build the image If it was on a mac you may need to rebuild it with ```docker build platform linux amd64```

Abhishek Agrawal · Answer

I built the image on my local and then I pushed to gcr I can do it again after adding dagster cloud install command

daniel · Answer

exec format error there usually indicates that it s trying to use a different system architecture than the one it was built on

Abhishek Agrawal · Answer

hmm that s strange I will try with dagster cloud first and see how I go I also think that it maybe a reason for this failure

daniel · Answer

I think if it was the dagster cloud thing there would be a different error there related to not being able to import that package

Abhishek Agrawal · Answer

ohh

Abhishek Agrawal · Answer

that exec error came when I try to get the logs the describe response didn t say much about the error though right

Abhishek Agrawal · Answer

in terms of image it s just built on my local and then pushed to gcr

daniel · Answer

I think the exec error is the contents of the logs it s saying the pod couldn t start up because the image wasn t built in a way that could be executed in the cluster

Abhishek Agrawal · Answer

ooh

daniel · Answer

Try building it with the flag I posted earlier to match the architecture of the k8s cluster

Abhishek Agrawal · Answer

got it will do it now

Abhishek Agrawal · Answer

while I do it here s the config file when I m adding the code locations ```location name spina image <http gcr io mutinex dev spina|gcr io mutinex dev spina> code source package name spina container context k8s image pull policy Always namespace dagster cloud run k8s config container config resources limits cpu 500m memory 1024Mi pod spec config node selector disktype ssd server k8s config container config resources limits cpu 100m memory 128Mi``` and here s the docker file which is pretty straightforward ```FROM python 3 9 0 WORKDIR app COPY spina requirements txt spina requirements txt ADD RUN pip3 install r spina requirements txt``` Does these look okay

daniel · Answer

that looks OK yeah I would suggest python 3 9 over python 3 9 0 so that you get the latest version with security patches

Abhishek Agrawal · Answer

thanks a ton < daniel> It works but now I need to solve something internally we were using JSON key files for GCP authentication so it s looking for that file

daniel · Answer

Here s one solution for that <https github com dagster io dagster discussions 12183>

Abhishek Agrawal · Answer

Hmmm that s pretty easy I will check with my infra guys Last I checked they were against using json key files I had k8s related question for the print statements I have in my code is it possible to view them in k8s pod somewhere Would you know how to access those

daniel · Answer

Is it possible to make a new post for new questions Our support oncall will see them and get them answered for that specific question the answer may depend on where exactly the print statement is

Abhishek Agrawal · Answer

Done