https://dagster.io/ logo
Powered by
Title
m

Mohammad Nazeeruddin

10/28/2021, 5:50 AM
Hi team, When we are creating pods using K8sRunLauncher in pipeline execution time. it's showing pod > status : NotReady. But pipeline executing successfully in this k8s pods i checked logs everything looks fine. we are deploying user-code-deployment helm chart with Argocd tool in this tool the k8s pods still in progress state only. How i can solve this pods NotReady status issue.
p

Prateek Agarwal

10/28/2021, 6:17 AM
Once the K8S pod executes the pipeline, should it be terminated/removed/deleted?
d

daniel

10/28/2021, 3:02 PM
Hi Mohammad, do you know what the 2 containers on the pod are? What does 'kubectl describe pod <pod name>' show? I was under the impression that dagster run pods typically only have 1 container, so it seems like your system is adding a second one and it's not starting for some reason
prateek this is the link I meant to post yesterday for your question: https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment#kubernetes-job-and-pod-ttl-management
m

Mohammad Nazeeruddin

10/29/2021, 8:38 AM
Hi @daniel, This is description content of the pod and containers are istio , user-code-deployment. So there is any issue with istio container ? is it possible to config istio with k8s pods? 
Name:         dagster-run-15ec8c3b-07ba-4a61-8c37-d856b6659964-bvnwf
Namespace:    dagster
Priority:     0
Node:         ip-10-11-225-106.ec2.internal/10.11.225.106
Start Time:   Thu, 28 Oct 2021 12:37:23 +0000
Labels:       <http://app.kubernetes.io/component=run_coordinator|app.kubernetes.io/component=run_coordinator>
              <http://app.kubernetes.io/instance=dagster|app.kubernetes.io/instance=dagster>
              <http://app.kubernetes.io/name=dagster|app.kubernetes.io/name=dagster>
              <http://app.kubernetes.io/part-of=dagster|app.kubernetes.io/part-of=dagster>
              <http://app.kubernetes.io/version=0.12.10|app.kubernetes.io/version=0.12.10>
              controller-uid=448e3e94-13dd-4b5a-801d-2702020d2ea4
              <http://istio.io/rev=default|istio.io/rev=default>
              job-name=dagster-run-15ec8c3b-07ba-4a61-8c37-d856b6659964
              <http://security.istio.io/tlsMode=istio|security.istio.io/tlsMode=istio>
              <http://service.istio.io/canonical-name=dagster|service.istio.io/canonical-name=dagster>
              <http://service.istio.io/canonical-revision=0.12.10|service.istio.io/canonical-revision=0.12.10>
Annotations:  <http://kubectl.kubernetes.io/default-container|kubectl.kubernetes.io/default-container>: dagster-run-15ec8c3b-07ba-4a61-8c37-d856b6659964
              <http://kubectl.kubernetes.io/default-logs-container|kubectl.kubernetes.io/default-logs-container>: dagster-run-15ec8c3b-07ba-4a61-8c37-d856b6659964
              <http://kubernetes.io/psp|kubernetes.io/psp>: nm.istio
              <http://prometheus.io/path|prometheus.io/path>: /stats/prometheus
              <http://prometheus.io/port|prometheus.io/port>: 15020
              <http://prometheus.io/scrape|prometheus.io/scrape>: true
              <http://sidecar.istio.io/status|sidecar.istio.io/status>:
                {"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istio-token","istiod-...
Status:       Running
IP:           100.112.31.41
IPs:
  IP:           100.112.31.41
Controlled By:  Job/dagster-run-15ec8c3b-07ba-4a61-8c37-d856b6659964
Init Containers:
  istio-init:
    Container ID:  <docker://d4762df662aa7b2194359f028c077c9aeb99c599cea8e1454578c3ae3624291>8
    Image:         <http://reg.ml.com:5000/istio/proxyv2:1.10.0|reg.ml.com:5000/istio/proxyv2:1.10.0>
    Image ID:      <docker-pullable://reg.ml.com:5000/istio/proxyv2@sha256:88c6c693e67a0f2492191a0e7d8020ddc85603bfc704f252655cb9eb5eeb3f58>
    Port:          <none>
    Host Port:     <none>
    Args:
      istio-iptables
      -p
      15001
      -z
      15006
      -u
      1337
      -m
      REDIRECT
      -i
      *
      -x

      -b
      *
      -d
      15090,15021,15020
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 28 Oct 2021 12:37:24 +0000
      Finished:     Thu, 28 Oct 2021 12:37:24 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:        100m
      memory:     128Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from dagster-token-nsgtq (ro)
Containers:
  dagster-run-15ec8c3b-07ba-4a61-8c37-d856b6659964:
    Container ID:  <docker://114fdad61b97662d14f73a5b203d1f2687fe4a473b7247c22f86b4fb3177edb>1
    Image:         <http://regt.nmlv.nml.com:5000/udp/infrastructure/dagsterimage0/user-code-example-v42:latest|regt.nmlv.nml.com:5000/udp/infrastructure/dagsterimage0/user-code-example-v42:latest>
    Image ID:      <docker-pullable://regt.ml.com:5000/udp/infrastructure/dagsterimage0/user-code-example-v42@sha256:400210f8be2d0bd9090bed959dc5aa74ec019af2cac58c5805c4f2ecc16ddc18>
    Port:          <none>
    Host Port:     <none>
    Args:
      dagster
      api
      execute_run
      {"__class__": "ExecuteRunArgs", "instance_ref": null, "pipeline_origin": {"__class__": "PipelinePythonOrigin", "pipeline_name": "hello_world_pipeline", "repository_origin": {"__class__": "RepositoryPythonOrigin", "code_pointer": {"__class__": "FileCodePointer", "fn_name": "hello_world_repo", "python_file": "/home/orchestrator/udp2/udp_repo.py", "working_directory": "/"}, "container_image": null, "executable_path": "/home/orchestrator/env/bin/python"}}, "pipeline_run_id": "15ec8c3b-07ba-4a61-8c37-d856b6659964"}
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 28 Oct 2021 12:37:25 +0000
      Finished:     Thu, 28 Oct 2021 12:37:27 +0000
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      dagster-pipeline-env  ConfigMap  Optional: false
    Environment:
      DAGSTER_HOME:         /home
      DAGSTER_PG_PASSWORD:  <set to the key 'postgresql-password' in secret 'rds-postgres-connection-password'>  Optional: false
    Mounts:
      /home/dagster.yaml from dagster-instance (rw,path="dagster.yaml")
      /var/run/secrets/kubernetes.io/serviceaccount from dagster-token-nsgtq (ro)
  istio-proxy:
    Container ID:  <docker://a640e1ba700df35d2120ac3bd79f45909e2b68c8882a8fde8386d38ce0c2e5c>0
    Image:         <http://reg.ml.com:5000/istio/proxyv2:1.10.0|reg.ml.com:5000/istio/proxyv2:1.10.0>
    Image ID:      <docker-pullable://reg.ml.com:5000/istio/proxyv2@sha256:88c6c693e67a0f2492191a0e7d8020ddc85603bfc704f252655cb9eb5eeb3f58>
    Port:          15090/TCP
    Host Port:     0/TCP
    Args:
      proxy
      sidecar
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --serviceCluster
      dagster-run-15ec8c3b-07ba-4a61-8c37-d856b6659964.dagster
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --log_output_level=default:info
      --concurrency
      2
    State:          Running
      Started:      Thu, 28 Oct 2021 12:37:26 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=3s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                    third-party-jwt
      PILOT_CERT_PROVIDER:           istiod
      CA_ADDR:                       istiod.istio-system.svc:15012
      POD_NAME:                      dagster-run-15ec8c3b-07ba-4a61-8c37-d856b6659964-bvnwf (v1:metadata.name)
      POD_NAMESPACE:                 dagster (v1:metadata.namespace)
      INSTANCE_IP:                    (v1:status.podIP)
      SERVICE_ACCOUNT:                (v1:spec.serviceAccountName)
      HOST_IP:                        (v1:status.hostIP)
      CANONICAL_SERVICE:              (v1:metadata.labels['<http://service.istio.io/canonical-name|service.istio.io/canonical-name>'])
      CANONICAL_REVISION:             (v1:metadata.labels['<http://service.istio.io/canonical-revision|service.istio.io/canonical-revision>'])
      PROXY_CONFIG:                  {}

      ISTIO_META_POD_PORTS:          [
                                     ]
      ISTIO_META_APP_CONTAINERS:     dagster-run-15ec8c3b-07ba-4a61-8c37-d856b6659964
      ISTIO_META_CLUSTER_ID:         Kubernetes
      ISTIO_META_INTERCEPTION_MODE:  REDIRECT
      ISTIO_METAJSON_ANNOTATIONS:    {"<http://kubernetes.io/psp|kubernetes.io/psp>":"nm.istio"}

      ISTIO_META_WORKLOAD_NAME:      dagster-run-15ec8c3b-07ba-4a61-8c37-d856b6659964
      ISTIO_META_OWNER:              <kubernetes://apis/batch/v1/namespaces/dagster/jobs/dagster-run-15ec8c3b-07ba-4a61-8c37-d856b6659964>
      ISTIO_META_MESH_ID:            cluster.local
      TRUST_DOMAIN:                  cluster.local
    Mounts:
      /etc/istio/pod from istio-podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from dagster-token-nsgtq (ro)
      /var/run/secrets/tokens from istio-token (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  istio-podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
      limits.cpu -> cpu-limit
      requests.cpu -> cpu-request
  istio-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  43200
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  dagster-instance:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      dagster-instance
    Optional:  false
  dagster-token-nsgtq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  dagster-token-nsgtq
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> for 300s
                 <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> for 300s
Events:          <none>
d

daniel

10/29/2021, 1:57 PM
That looks like a problem with the istio-proxy container which isn't set by dagster - so this may be more of an istio question than a dagster question. If you're able to to determine what additional config on the job would help make it ready, we can help with determining how to set that config on the dagster job - but I don't really have the istio knowledge to diagnose this
👍 2
m

Mohammad Nazeeruddin

10/29/2021, 2:32 PM
Okay Thanks @daniel and Is it possible to config annotations ,livenessProbe and readiness in dagster k8s job ?
@daniel Anything on this ^
d

daniel

11/01/2021, 1:34 PM
there's an annotations example here: https://docs.dagster.io/deployment/guides/kubernetes/customizing-your-deployment I don't think we currently support setting a custom liveness probe or readiness probe
m

Mohammad Nazeeruddin

11/02/2021, 9:55 AM
Configured istio in customising deployment like this, still pod in NotReady state. 
@solid(
    tags={
        'dagster-k8s/config': {
            'container_config': {
                'resources': {
                    'requests': {'cpu': '250m', 'memory': '64Mi'},
                    'limits': {'cpu': '500m', 'memory': '2560Mi'},
                }
            },
            'pod_template_spec_metadata': {
                'annotations': {"<http://sidecar.istio.io/inject|sidecar.istio.io/inject>": "false"}  <istio enabled false>
            },
        },
    },
)
is it work ? with istio.
d

daniel

11/02/2021, 12:20 PM
I can help with questions about dagster but not about the details of istio, so unfortunately I’m not sure
m

Mohammad Nazeeruddin

11/02/2021, 12:24 PM
'pod_template_spec_metadata': {
                'annotations': {"<http://sidecar.istio.io/inject|sidecar.istio.io/inject>": "false"}  <istio enabled false>
            },
I want to know if is it possible or not with dagster ? dagster support these istio in solids. that is my question @daniel could you please help us.
d

daniel

11/02/2021, 12:29 PM
I’d expect that annotation to appear in the dagster pod, yeah. Is it appearing when you describe the pod?
m

Mohammad Nazeeruddin

11/02/2021, 1:38 PM
<http://sidecar.istio.io/status|sidecar.istio.io/status>:
                {"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istio-token","istiod-..
it's showing this one but what i configured in solid it's not showing in pod.
d

daniel

11/02/2021, 1:46 PM
Can you paste the whole description of the pod?
👍 1
When you say ‘that I configured in the solid’ - you would want to configure it on the pipeline, not the solid, for it be applied to the dagster-run pod
Configure the tag I mean)
Pipelines can have tags just like solids
👍 1
m

Mohammad Nazeeruddin

11/02/2021, 1:52 PM
I am using dagster Version 0.12.10. no issue with version right? for these configs.
i will try with pipeline tag. Thank you @daniel.
Thank you so much @daniel, it's working with pipeline tag.👍
❤️ 1
:condagster: 2
3 Views
#dagster-supportJoin Slack
Powered by