max
11/09/2021, 12:44 AMbitsofinfo
12/08/2021, 8:32 PMmax
12/15/2021, 4:08 PMCorey Capel
12/15/2021, 7:49 PM404 - Internal Server Error
on https://<our-org-id>.dagster.cloud/auth/saml/consume
. If I manually navigate to https://<our-org-id>.dagster.cloud
afterward and click the SSO Sign-in button, things work fine. This has been happening for a few days.Corey Capel
12/15/2021, 8:12 PMdagster-run
task throws the following error:
dagster.check.CheckError: Failure condition: Couldn't import module dagster_cloud when attempting to load the class dagster_cloud.DagsterCloudAgentInstance
Do we need to be including Dagster Cloud specific modules in our pipeline code?Corey Capel
12/15/2021, 8:20 PMException: Failure loading server endpoint for location hello_cereal: Exception: Timed out waiting for server hello_cereal_7017efd.dagster-agent-<our-org-id>-prod.local:4000
Stack Trace:
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 411, in _reconcile
new_updated_endpoint: GrpcServerEndpoint = next(update_gens[to_update_key])
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 508, in _gen_update_server
updated_server = self._create_new_server_endpoint(location_name, new_metadata)
File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 106, in _create_new_server_endpoint
server_id = self._wait_for_server(host=service.hostname, port=4000, timeout=60)
File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 535, in _wait_for_server
raise Exception(f"Timed out waiting for server {host}:{port}")
That look familiar to anyone? The last successful run was Nov 15th.Prratek Ramchandani
01/03/2022, 2:57 PMLyudmil Nenov
01/24/2022, 4:08 PMRohan Kshirsagar
02/04/2022, 5:59 PM```❯ dagster-cloud agent run
2022-02-03 102054 -0800 - dagster_cloud - INFO - Starting Dagster Cloud agent...
2022-02-03 102054 -0800 - dagster_cloud - INFO - Waiting for workspace to be ready...
2022-02-03 102059 -0800 - dagster_cloud - INFO - Started polling for requests from https://rohan.agent.dagster.cloud```I kept this ^ running in a terminal
```❯ dagster-cloud config setup
? How would you like to authenticate the CLI? Authenticate using token or keep current settings
? Dagster Cloud organization: rohan
? Dagster Cloud user token: *************************************
? Default deployment: prod```However, when I try to add a location to my workspace I run into this issue
❯ dagster-cloud workspace add-location unify_news_dag --package-name unify_news_dag --deployment prod
Error: ('Unable to add/update code location: ', 'dagster.core.errors.DagsterUserCodeUnreachableError: Could not send request to agent since no agents have recently heartbeated\n')
Rohan Kshirsagar
02/05/2022, 12:31 AMWaiting for agent to sync changes to unify-news-dag...
Error: Some locations failed to load after being synced by the agent:
Error loading unify-news-dag: {'__typename': 'PythonError', 'message': "dagster.core.errors.DagsterImportError: Encountered ImportError: `No module named 'pandas'` while importing module unify_news_dag. Local modules were resolved using the working directory `/Users/rohan/projects/unify-news-dag`. If another working directory should be used, please explicitly specify the appropriate path using the `-d` or `--working-directory` for CLI based targets or the `working_directory` configuration option for workspace targets. \n", 'stack': [' File "/Users/rohan/miniconda3/envs/unify-news-dag/lib/python3.8/site-packages/dagster/grpc/server.py", line 216, in __init__\n self._repository_symbols_and_code_pointers.load()\n', ' File "/Users/rohan/miniconda3/envs/unify-news-dag/lib/python3.8/site-packages/dagster/grpc/server.py", line 92, in load\n self._loadable_repository_symbols = load_loadable_repository_symbols(\n', ' File "/Users/rohan/miniconda3/envs/unify-news-dag/lib/python3.8/site-packages/dagster/grpc/server.py", line 110, in load_loadable_repository_symbols\n loadable_targets = get_loadable_targets(\n', ' File "/Users/rohan/miniconda3/envs/unify-news-dag/lib/python3.8/site-packages/dagster/grpc/utils.py", line 47, in get_loadable_targets\n else loadable_targets_from_python_package(package_name, working_directory)\n', ' File "/Users/rohan/miniconda3/envs/unify-news-dag/lib/python3.8/site-packages/dagster/core/workspace/autodiscovery.py", line 31, in loadable_targets_from_python_package\n module = load_python_module(\n', ' File "/Users/rohan/miniconda3/envs/unify-news-dag/lib/python3.8/site-packages/dagster/core/code_pointer.py", line 130, in load_python_module\n raise DagsterImportError(\n']}
Error:
Rohan Kshirsagar
02/05/2022, 6:08 PMEvan Arnold
02/08/2022, 4:59 PMdeployments
? My initial framework was a Heroku/12-Factor sort of pattern. Meaning, I thought of a code as a consistent chunk, deployed identically to every environment, with secrets & external services injected via environment variables.
Obviously, that doesn't directly apply here. As far as I can tell, Dagster Cloud is completely agnostic around secrets management? And external dependencies are bound to graphs to make jobs in code. So what then is a deployment
? Is it a "place where your code is running"? Should I think of it as a library of potential destinations? Or... what?
Thanks in advance for any guidance!Lyudmil Nenov
02/08/2022, 5:48 PMLyudmil Nenov
02/10/2022, 5:22 PMpythonLogs:
## The names of python loggers that will be captured as Dagster logs
managedPythonLoggers:
- de
## The log level for the instance. Logs emitted below this severity will be ignored.
## One of [NOTSET, DEBUG, INFO, WARNING, WARN, ERROR, FATAL, CRITICAL]
pythonLogLevel: INFO
in helm show values dagster/dagster
chart valuesEvan Arnold
02/11/2022, 10:34 PMEvan Arnold
02/14/2022, 9:29 PMEvan Arnold
02/14/2022, 9:50 PMBennett Norman
02/24/2022, 2:57 AMSeth Kimmel
02/24/2022, 3:16 AMCorey Capel
03/03/2022, 8:42 PMprod
to dev
) or is it simpler to delete existing workspaces, create the new dev
deployment, and re-deploy workspaces to the new deployment?Evan Arnold
03/10/2022, 5:13 PM• If you're using the Kubernetes, ECS, or Docker agents, you'll need to package your code into a Docker image and push it to a registry that your agent can access. Because of Dagster Cloud's hybrid architecture, Dagster's hosted components don't need access to your image - only your agent needs to be able to pull your image. The Dockerfile for your image does not need to specify an entry point or command - those will be specified by your agent when it runs code using your supplied image.I'm curious about how one implements the part in bold
Stephen Bailey
03/14/2022, 3:08 PMZach
03/14/2022, 3:51 PMLiang Zhang
03/15/2022, 10:53 AMStephen Bailey
03/21/2022, 2:48 PMlocations.yml
files and multiple actions maybe...Nikhaar Gupta
03/22/2022, 4:40 PMCharlie Bini
04/01/2022, 5:15 PMNikhaar Gupta
04/05/2022, 5:26 PMTraceback (most recent call last):
File "/opt/python/lib/python3.10/code.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 116, in <module>
File "<input>", line 103, in _create_marks_df
File "<input>", line 274, in _barclays
File "/opt/python/lib/python3.10/site-packages/tabula/io.py", line 322, in read_pdf
output = _run(java_options, kwargs, path, encoding)
File "/opt/python/lib/python3.10/site-packages/tabula/io.py", line 91, in _run
raise JavaNotFoundError(JAVA_NOT_FOUND_ERROR)
tabula.errors.JavaNotFoundError: `java` command is not found from this Python process.Please ensure Java is installed and PATH is set for `java`
I'm wondering if anyone knows a good approach to utilize tabula for dagster jobs? Here are the options that come to mind:
• Install both java and python into the docker file. However, from reading online, it sounds like docker is meant to utilize one service per container: https://docs.docker.com/config/containers/multi-service_container/ - "It is generally recommended that you separate areas of concern by using one service per container"
• Use a separate container for Java
• Use a different python package that does not have other dependenciesCharlie Bini
04/07/2022, 6:07 PMCharlie Bini
04/08/2022, 4:49 PM