https://dagster.io/ logo
Powered by
#dagster-support
Title
# dagster-support
f

Fraser Marlow

01/20/2023, 3:03 PM
d

daniel

01/20/2023, 3:23 PM
@Akshay Verma try this section and see if it helps: https://docs.dagster.io/deployment/guides/running-locally#locating-your-code
(the "locating your code" section)
❤️ 1
a

Akshay Verma

01/20/2023, 3:26 PM
Hi @daniel It is behaving like described in the docs. 'Without command line arguments' it is looking for pyproject.toml file. I was wondering If I can run just dagit and dagster-daemon and then later add a pipeline module from another folder location.
That way I might be able to connect multiple dagster pipelines to a single dagit instance?
d

daniel

01/20/2023, 3:29 PM
If you use a workspace file (https://docs.dagster.io/concepts/code-locations/workspace-files#workspace-files) - you can point 
dagster dev
at that file, then change the contents of the workspace.yaml file later and reload the workspace from within dagit on the Code Locations page by pressing the "Reload all" button
❤️ 1
a

Akshay Verma

01/20/2023, 3:30 PM
Great, will give that a try
Hi, I tried adding a blank 
workspace.yaml
file and then starting dagster using 
dagster dev
but I get the following error
Copy code
(dagster) akshay_verma@shazam-102-orchestrate:~$ dagster dev -w ~/dagster/dagster_home/workspace.yaml 
2023-01-20 15:33:11 +0000 - dagster - INFO - Using temporary directory /home/akshay_verma/tmpihwblgz4 for storage. This will be removed when dagster dev exits.
2023-01-20 15:33:11 +0000 - dagster - INFO - To persist information across sessions, set the environment variable DAGSTER_HOME to a directory to use.
2023-01-20 15:33:11 +0000 - dagster - INFO - Launching Dagster services...
Traceback (most recent call last):
  File "/home/akshay_verma/dagster_deploy/dagster/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/home/akshay_verma/dagster_deploy/dagster/lib/python3.9/site-packages/dagster/_cli/__init__.py", line 46, in main
    cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
  File "/home/akshay_verma/dagster_deploy/dagster/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/akshay_verma/dagster_deploy/dagster/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/akshay_verma/dagster_deploy/dagster/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/akshay_verma/dagster_deploy/dagster/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/akshay_verma/dagster_deploy/dagster/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/akshay_verma/dagster_deploy/dagster/lib/python3.9/site-packages/dagster/_cli/dev.py", line 103, in dev_command
    dagit_process = open_ipc_subprocess(
  File "/home/akshay_verma/dagster_deploy/dagster/lib/python3.9/site-packages/dagster/_serdes/ipc.py", line 195, in open_ipc_subprocess
    check.list_param(parts, "parts", str)
  File "/home/akshay_verma/dagster_deploy/dagster/lib/python3.9/site-packages/dagster/_check/__init__.py", line 781, in list_param
    return _check_iterable_items(obj, of_type, "list")
  File "/home/akshay_verma/dagster_deploy/dagster/lib/python3.9/site-packages/dagster/_check/__init__.py", line 1769, in _check_iterable_items
    raise CheckError(
dagster._check.CheckError: Member of list mismatches type. Expected <class 'str'>. Got ('/home/akshay_verma/dagster/dagster_home/workspace.yaml',) of type <class 'tuple'>.
I am using this example https://docs.dagster.io/concepts/code-locations/workspace-files#loading-workspace-files
d

daniel

01/20/2023, 3:38 PM
I think this may be an issue with the new 'dev' command when a workspace file is passed in. We'll get that fixed shortly, but I would expect dagit -w and dagster-daemon -w to work in the meantime.
👍 1
thanks for the report
a

Akshay Verma

01/20/2023, 3:41 PM
dagit reads but complains of empty yaml
Copy code
dagster._check.CheckError: Invariant failed. Description: Could not parse a workspace config from the yaml file at /home/akshay_verma/dagster/dagster_home/workspace.yaml. Check that the file contains valid yaml.
d

daniel

01/20/2023, 3:41 PM
What are the contents of your workspace.yaml file?
a

Akshay Verma

01/20/2023, 3:41 PM
blank, no contents
because I wanted to run just dagit instance
d

daniel

01/20/2023, 3:42 PM
this should work
Copy code
load_from: []
a

Akshay Verma

01/20/2023, 3:45 PM
That works. Thanks
d

daniel

01/20/2023, 3:59 PM
dagster dev should also work if you run it in the same folder as the workspace.yaml (with no arguments)
D 2
a

Akshay Verma

01/23/2023, 2:58 PM
I am able to run latest dagster dev successfully. Only thing I am not able to figure out is, how to put 'more than one' pipeline into the 
workspace.yaml
file. Currently I am using a checked out version, but if it is multiple pipelines then they would have to be installed?
d

daniel

01/23/2023, 2:59 PM
Can you explain more what you mean by 'a checked out version' and 'installed'? Maybe an example would help
a

Akshay Verma

01/23/2023, 3:01 PM
Checked out - source code checked out from repo in folders only, not installed in python env installed - installed as Python module in the dagster python env and thus can use 'load from module' (correct?)
d

daniel

01/23/2023, 3:04 PM
Both can be loaded as modules actually - python just needs to be able to import them
Which it can also do even if the package isn’t installed by looking in folders relative to the current directory
a

Akshay Verma

01/23/2023, 3:08 PM
to provide relative (or absolute) location of multiple folders, does the yaml look like this?
Copy code
load_from:
  - python_package: example1
  - python_package: example2
d

daniel

01/23/2023, 3:09 PM
yeah (you can also use python_module, they're the same and the latter is a bit more accurate for exactly this reason) That assumes that there are example1 and example2 folders that are child folders of the folder with the workspace.yaml folder in it
D 1
a

Akshay Verma

01/23/2023, 3:11 PM
ahh yes, and with location , where example1 and example2 are folders with python code, like this?
Copy code
load_from:
  - python_package: /home/user/example1
  - python_package: /home/user/example2
d

daniel

01/23/2023, 3:12 PM
you might want this to make that work
Copy code
load_from:
  - python_module:
      module_name: example1
      working_directory: /home/user
  - python_module:
      module_name: example2
      working_directory: /home/user
🌈 1
a

Akshay Verma

01/23/2023, 3:15 PM
Great, I was not able to find this skeleton/example in the docs. you get this sense from https://docs.dagster.io/concepts/code-locations#cloud-deployment , but was not able to find anyhting similar for OSS deployments using workspace.yaml file.
d

daniel

01/23/2023, 3:15 PM
2 Views
Powered by