https://dagster.io/ logo
Join Slack
Powered by
Good day! Do you all have a list of static IPs so ...
# ask-community
a
Good day! Do you all have a list of static IPs so that we can whitelist them?
Bumping in the event these threads get lost over time.
b
our IPs for 
dagster.cloud
are currently:
Copy code
✗ dig dagster.cloud +short
35.84.161.144
35.85.173.250
44.233.98.242
54.189.250.39
note that we don't have a strong guarantee that these will not change
a
@Matthew Seal FYI on non-strong guarantees.
@ba do you offer VPC peering or have another recommended approach to allow Dagster access to private clusters?
b
using the hybrid deployment model is usually recommended if you need the compute to access resources in a restricted context
a
Thank you! We’ll discuss this option.
Happy Friday! We are still working on setting up our Dagster POC. Initially, we went down the cloud path, but due to having restricted environments we need to migrate to a hybrid. I’m following the Getting Started with Dagster Cloud using a Hybrid Deployment documentation, and am stuck on the step that says: 
When prompted, click Select under Hybrid deployment
. I do not see anywhere to modify an existing deployment to be hybrid. When I removed our 
prod
deployment, and created a new one, there was no prompt or option to select Hybrid. Is there a different approach other than what is documented?
r
we can manually disable your serverless agent - cc @Joe
a
Thank you. When that is complete should I be able to run the 
dagster-io/dagster-cloud-hybrid-quickstart
to connect the hybrid workspace?
We have the agent running in our k8s cluster already
j
yep as soon as i deactivate your serverless agent your hybrid one will take its place
whats your org_id?
a
noteable
j
@Anders Berg you should be using your hybrid agent now
a
Thank you! I tried to rerun the 
Build and Deploy to Dagster Cloud Hybrid
job and got the following errors in the agent within our cluster:
Copy code
2022-08-26 00:35:23 +0000 - dagster_cloud.agent - INFO - Received request [058c52e2-40d6-4f27-bc8e-da6c22493444: DagsterCloudApi.CHECK_FOR_WORKSPACE_UPDATES].
2022-08-26 00:35:24 +0000 - dagster_cloud.agent - INFO - Finished processing request [058c52e2-40d6-4f27-bc8e-da6c22493444: DagsterCloudApi.CHECK_FOR_WORKSPACE_UPDATES].
2022-08-26 00:35:24 +0000 - dagster_cloud.agent - INFO - Uploading response for request [058c52e2-40d6-4f27-bc8e-da6c22493444: DagsterCloudApi.CHECK_FOR_WORKSPACE_UPDATES].
2022-08-26 00:35:24 +0000 - dagster_cloud.agent - INFO - Finished uploading response for request [058c52e2-40d6-4f27-bc8e-da6c22493444: DagsterCloudApi.CHECK_FOR_WORKSPACE_UPDATES].
2022-08-26 00:35:29 +0000 - dagster_cloud.user_code_launcher - INFO - Reconciling to reach {}. To add: {}. To update: {}. To remove: {(prod, example_hybrid_location, 1661473667.268101)}. To upload: {}.
2022-08-26 00:35:29 +0000 - dagster_cloud.user_code_launcher - INFO - Removing server for prod:example_hybrid_location
2022-08-26 00:35:29 +0000 - dagster_cloud.user_code_launcher - INFO - Finished reconciling in 0.028425931930541992 seconds.
2022-08-26 13:25:47 +0000 - dagster_cloud.agent - ERROR - Caught error:
dagster_cloud_cli.core.errors.DagsterCloudHTTPError: Unexpected GraphQL response: 

Stack Trace:
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 187, in run_loop
    for error in self.run_iteration(instance, user_code_launcher):
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 844, in run_iteration
    result = instance.graphql_client_for_deployment(deployment_name).execute(
  File "/dagster-cloud-cli/dagster_cloud_cli/core/graphql_client.py", line 56, in execute
    return self._execute_retry(query, variable_values)
  File "/dagster-cloud-cli/dagster_cloud_cli/core/graphql_client.py", line 95, in _execute_retry
    raise DagsterCloudHTTPError(http_error) from http_error

The above exception was caused by the following exception:
requests.exceptions.HTTPError: Unexpected GraphQL response

Stack Trace:
  File "/dagster-cloud-cli/dagster_cloud_cli/core/graphql_client.py", line 92, in _execute_retry
    raise requests.HTTPError("Unexpected GraphQL response", response=response)

2022-08-26 13:33:38 +0000 - dagster_cloud.agent - ERROR - Failed to check for workspace updates: 
dagster_cloud_cli.core.errors.DagsterCloudHTTPError: Unexpected GraphQL response: 

Stack Trace:
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 214, in run_loop
    self._check_update_workspace(instance, user_code_launcher)
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 238, in _check_update_workspace
    self._query_for_workspace_updates(instance, user_code_launcher)
  File "/dagster-cloud/dagster_cloud/agent/dagster_cloud_agent.py", line 440, in _query_for_workspace_updates
    result = instance.organization_scoped_graphql_client().execute(
  File "/dagster-cloud-cli/dagster_cloud_cli/core/graphql_client.py", line 56, in execute
    return self._execute_retry(query, variable_values)
  File "/dagster-cloud-cli/dagster_cloud_cli/core/graphql_client.py", line 95, in _execute_retry
    raise DagsterCloudHTTPError(http_error) from http_error

The above exception was caused by the following exception:
requests.exceptions.HTTPError: Unexpected GraphQL response

Stack Trace:
  File "/dagster-cloud-cli/dagster_cloud_cli/core/graphql_client.py", line 92, in _execute_retry
    raise requests.HTTPError("Unexpected GraphQL response", response=response)
In the GitHub action logs I do see some maybe interesting data: 
'organization': None,
and 
url = '<https://dagster.cloud/>'
. I’m wondering if these need to be configured further to specify the organization as 
noteable
and the url to 
noteable.dagster.cloud
?
I haven’t done a deep dive into what the CI pipeline is doing, but I’m assuming that if the agent in our cluster picked up the job, that the agent token was enough to associate the run with our account….?
Adding 
DAGSTER_CLOUD_URL
env to CI might have helped, within the CI pipeline I receive the following errors now:
Copy code
Deploying location example_hybrid_location to deployment 922518e836a165675944f154b437a4714343a5f9...
Error: 404 Client Error: Not Found for url: ***/922518e836a165675944f154b437a4714343a5f9/graphql: Not Found
j
@Anders Berg im not sure what guide you're following for setting up ci but the docs in https://github.com/dagster-io/dagster-cloud-hybrid-quickstart#set-up-secrets might be useful
DAGSTER_CLOUD_API_TOKEN
and 
ORGANIZATION_ID
is likely what you need
a
I have those:
j
are you using the quickstart repo for hybrid?
a
It fails on this CI command
Copy code
Run dagster-io/dagster-cloud-action/actions/utils/deploy@main
  with:
    organization_id: noteable
    pr: 1
    pr_status: open
    location: {
    "name": "example_hybrid_location",
    "directory": "./",
    "build_folder": "./",
    "registry": "<http://ghcr.io/noteable-io/dagster-integration-hybrid|ghcr.io/noteable-io/dagster-integration-hybrid>",
    "location_file": "dagster_cloud.yaml"
  }
    image_tag: d6a167a5cf013eae8f65f1cdc0d97b8cf0e4d91b
  env:
    DAGSTER_CLOUD_URL: ***
    GITHUB_TOKEN: ***
    DAGSTER_CLOUD_API_TOKEN: ***
With the following output:
Copy code
Deploying location example_hybrid_location to deployment 922518e836a165675944f154b437a4714343a5f9...
Error: 404 Client Error: Not Found for url: ***/922518e836a165675944f154b437a4714343a5f9/graphql: Not Found
Error: Deploy failed. To view status of your code locations, visit ***/922518e836a165675944f154b437a4714343a5f9/workspace
j
hmmm is this still happening? if it'd be useful maybe we can hop on a zoom?
a
Yes, it is still happening. I’m available whenever you are.
j
ok give me a few minutes
5 Views
Open in Slack
PreviousNext
Powered by