https://dagster.io/ logo
Join Slack
Powered by
Hey Dagster, I have a question about the docker de...
# ask-community
a
Hey Dagster, I have a question about the docker deploy example here - https://github.com/dagster-io/dagster/blob/1.1.17/examples/deploy_docker/docker-compose.yml#L16-L67 - How would this look if I wanted to use a workspace or a module instead of a single repo.py file?
🤖 1
d
Hi aaron - the example actually uses a workspace.yaml file: https://github.com/dagster-io/dagster/blob/1.1.17/examples/deploy_docker/workspace.yaml What code the server is using is here: https://github.com/dagster-io/dagster/blob/1.1.17/examples/deploy_docker/Dockerfile_user_code#L23 You could reference a module there instead with '-m' instead of a file with '-f'
a
Thanks for the reply @daniel - What is the gRPC server for? I thought that is for hosting my code files. I mount my module src into the 
opt/dagster/app
folder on the code server, is that the right way to go about this? In production would we just copy the module over directly?
d
the gRPC server is for loading your Dagster code, yeah
Mounting a volume is one way to get your code available, yeah - there is some guidance on how to do that here: https://docs.dagster.io/deployment/guides/docker#mounting-volumes
a
Thanks, I have been reading that. I am at a place now where I have my compose building and starting the code server, daemon and dagit. That being said, I cannot see any of my jobs in dagit. If I try adding the 
/opt/dagster/app
as the 
-m
module the code server throws an error saying there is no module named. Maybe it has something to do with my module structure within the code server
d
are you able to load the jobs locally in dagit?
a
yes, let me run again to confirm
d
running "python -m <module name>" in the same folder as the grpc server would be pretty similar to what the grpc server is doing under the hood, and could help rule out dagster-specific issues
👍 1
a
let me give that a try - I have python 3.11 installed locally so have to run it all through docker because gRPC can't install on 3.11
I think I understand it better now. Just running into the issue of how do I install the module. Is there a way I can just set the working directory?
d
there's a --working-directory argument, yeah
that determines the root folder for local imports in your python code
you don't neccesarily have to install the module, it just has to be able to load in python
a
ahh, so in this case I would just need to make sure my requirements are installed first
let me try this
I keep getting the below error - does that mean it can't load the module?
Copy code
File "/usr/local/lib/python3.10/site-packages/dagster/_check/__init__.py", line 1699, in failed
    raise CheckError(f"Failure condition: {desc}")
dagster._check.CheckError: Failure condition: invalid
d
do you have a full stack trace?
a
Copy code
root@e2c0c23a4b72:/opt/dagster/app# dagster api grpc -h 0.0.0.0 -p 4000 --working-directory ./data_pipeline_dagster
Traceback (most recent call last):
  File "/usr/local/bin/dagster", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/dagster/_cli/__init__.py", line 46, in main
    cli(auto_envvar_prefix=ENV_PREFIX)  # pylint:disable=E1123
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/dagster/_cli/api.py", line 702, in grpc_command
    server = DagsterGrpcServer(
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/server.py", line 916, in __init__
    self._api_servicer = DagsterApiServer(
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/server.py", line 256, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/server.py", line 114, in __init__
    loadable_targets = get_loadable_targets(
  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/utils.py", line 60, in get_loadable_targets
    check.failed("invalid")
  File "/usr/local/lib/python3.10/site-packages/dagster/_check/__init__.py", line 1699, in failed
    raise CheckError(f"Failure condition: {desc}")
dagster._check.CheckError: Failure condition: invalid
d
Ah that's a bad error message, but you also need to include a module or a python file
working directory tells it what base folder to use for local imports, but doesn't tell it what module or file to try to import
a
so the command also needs -m?
d
Yeah
Do you have a corresponding python command that's working that you want to map onto the dagster command?
a
I do not, we have just been running the daemon locally from the root folder and testing using the dagit ui
d
How have you been telling the daemon which code to load?
a
haven't been passing any parameters. Just 
dagster-daemon run
d
for that to work, there would need to either be a pyproject.toml or a workspace.yaml in the same folder that tells it what code to load
a
let me show you my folder structure, maybe that will help
d
I think step 1 is getting the code to load via 
python
- at that point, getting it to load in 
dagster api grpc
should be straighforward
a
I see, so once I can run 
python -m <module>
it should load through grpc
d
yeah
a
ok, so let me work through that
Running 
python -m
when I am loading my resources 
__init__.py
it can't find the environment variables. I have a 
.env
file, but with docker compose would most likely use docker environment variables
ok, fixed the errors, but now I am getting 
'data_pipeline_dagster' is a package and cannot be directly executed
does it need to be executed?
d
this is when running python? or dagster?
a
python
d
what's the command exactly?
a
python -m data_pipeline_dagster
d
what version of python are you using?
a
locally 3.11.0, in the containers 3.10.9
I can try from inside the code container
d
might be a 3.11 thing then - you could run 'import data_pipeline_dagster' inside a Python shell then
e.g.
Copy code
python
>>> import data_pipeline_dagster
a
Copy code
\AppData\Roaming\Python\Python311\site-packages\dagster\_core\definitions\input.py:563: ExperimentalWarning: "asset_key" is an experimental argument to function "InputDefinition.__init__". It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at <https://docs.python.org/3/library/warnings.html#describing-warning-filters>.  return InputDefinition(
C:\Users\atawil\AppData\Roaming\Python\Python311\site-packages\dagster\_core\definitions\input.py:316: ExperimentalWarning: "asset_key" is an experimental argument to function "InputDefinition.__init__". It may break in future versions, even between dot releases. To mute warnings for experimental functionality, invoke warnings.filterwarnings("ignore", category=dagster.ExperimentalWarning) or use one of the other methods described at <https://docs.python.org/3/library/warnings.html#describing-warning-filters>.  return InputDefinition(
let me try the same from the container
d
if there are no errors that's good
❤️ 1
a
yes, just looks like warnings
d
then i would expect 'dagit -m data_pipeline_dagster' to work or 'dagster api grpc -m data_pipeline_dagster -p 4000' to work too
a
let me see
it's working now!
Some issues with the module itself causing the code server to not initialize properly. Now I just have to clean it up
really appreciate the help Daniel!
condagster 1
203 Views
Open in Slack
PreviousNext
Powered by