https://dagster.io/ logo
Join Slack
Powered by
Hi all, I am trying to set up my Dagster Cloud ser...
# dagster-plus
j
Hi all, I am trying to set up my Dagster Cloud serverless. I want to use Dagster Cloud from my Gitlab repo using the CLI (I am following the instructions here: https://docs.dagster.io/dagster-cloud/deployment/serverless#without-github-gitlab-bitbucket-or-local-development). I started by running the "assets_modern_data_stack" creation from example and it appears in my repo. I try to deploy the python executable with: 
dagster-cloud serverless deploy-python-executable \
--location-name quickstart_mds \
--package-name assets_modern_data_stack
And I always get the error: `Error: Could not find a 
setup.py
or 
requirements.txt
in the target directory. You must specify your required Python dependencies (including the 
dagster-cloud
package) along with your source files to deploy to Dagster Cloud.` I do have the file 
setup.py
in the project directory and I made sure to include 
install_requires=["dagster_cloud"]
after installing. Is there something else I am missing? Thanks!
s
Hi James, the 
deploy-python-executable
should be run in the same directory that contains 
setup.py
. Could you check you are in your project directory? Alternatively, you can also pass in the target directory as an argument to 
deploy-python-executable
, for example 
dagster-cloud serverless deploy-python-executable ./my-project --location-name ...
Let us know if this helps. We will update our docs to make this clearer.
j
Thanks! I got further than before. I got the following error: 
*FileNotFoundError: [*Errno *2]* No such file or directory: 'python'
s
Can share the full log for this error please?
Also can you share which environment you see this error on - gitlab or your local workstation?
j
It is pretty log
Hang on... it is too long for Slack to copy paste
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ args = 
*[*                                 │ │
│ │ │ 'python', │ │ │ │ │ '/Users/jamesrosenthal/chinook/dagster-etl/setup.py', │ │ │ │ │ 'egg_info', │ │ │ │ │ │ │ │ │ '--egg-base=/var/folders/v1/gyd4_prs14nbbgh9hm306whr0000gn/T/tmph… │ │ │ │ 
*]*                                 │ │
│ │ c2pread = 3 │ │ │ │ c2pwrite = 4 │ │ │ │ child_exception_type = 
*<class* 'OSError'*>*                         │ │
│ │ child_exec_never_called = False │ │ │ │ close_fds = True │ │ │ │ creationflags = 0 │ │ │ │ cwd = None │ │ │ │ env = None │ │ │ │ env_list = None │ │ │ │ err_filename = 'python' │ │ │ │ err_msg = 'No such file or directory' │ │ │ │ errno_num = 2 │ │ │ │ errpipe_data = 
*bytearray(*b'OSError:2:'*)*                      │ │
│ │ errpipe_read = 7 │ │ │ │ errpipe_write = 8 │ │ │ │ errread = 5 │ │ │ │ errwrite = 6 │ │ │ │ exception_name = 
*bytearray(*b'OSError'*)*                       │ │
│ │ executable = b'python' │ │ │ │ executable_list = 
*(*                                 │ │
│ │ │ b'/opt/homebrew/bin/python', │ │ │ │ │ b'/opt/homebrew/sbin/python', │ │ │ │ │ b'/Applications/SnowSQL.app/Contents/MacOS/python', │ │ │ │ │ b'/opt/homebrew/bin/python', │ │ │ │ │ b'/opt/homebrew/sbin/python', │ │ │ │ │ b'/usr/local/bin/python', │ │ │ │ │ b'/System/Cryptexes/App/usr/bin/python', │ │ │ │ │ b'/usr/bin/python', │ │ │ │ │ b'/bin/python', │ │ │ │ │ b'/usr/sbin/python', │ │ │ │ │ ... +3 │ │ │ │ 
*)*                                 │ │
│ │ fds_to_keep = 
*{*8*}*                                │ │
│ │ gid = None │ │ │ │ gids = None │ │ │ │ hex_errno = 
*bytearray(*b'2'*)*                          │ │
│ │ low_fds_to_close = 
*[]*                                 │ │
│ │ orig_executable = 'python' │ │ │ │ p2cread = -1 │ │ │ │ p2cwrite = -1 │ │ │ │ part = b'' │ │ │ │ pass_fds = 
*()*                                 │ │
│ │ pid = 18345 │ │ │ │ preexec_fn = None │ │ │ │ restore_signals = True │ │ │ │ self = 
*<Popen:* returncode: 255 args: *[*'python',              │ │
│ │ '/Users/jamesrosenthal/chinook/dags...
*>*              │ │
│ │ shell = False │ │ │ │ start_new_session = False │ │ │ │ startupinfo = None │ │ │ │ sts = 65280 │ │ │ │ uid = None │ │ │ │ umask = -1 │ │ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯
That is the last piece
s
Thank you. In the meantime you can try switching the command 
deploy-python-executable
to 
deploy-docker
. If you have docker installed locally this may help. It may not work if you're running this in gitlab.
j
Thanks-- would I need to have a DockerFile in the directory for that to work?
I already have Docker installed and running on my laptop
And it is my local workstation
To answer your earlier question
Running with 
deploy-docker
now
s
No that command will create a dockerfile for you and use that to build and deploy. For the 
deploy-python-executable
error, it may help to ensure you have a python executable available in your PATH. Are you able to just run 
python
from the same terminal? It does look like our bug where we should find and use the same python interpreter that is being used for the 
dagster-cloud
.
j
It seems like it sees my setup file... that's a good sign!
I can run python3
s
A temporary workaround would be to also link 
python3
as 
python
. I'll file a bug and work on a fix on our end. If 
deploy-docker
works you can continue using that for now.
j
It worked!! 🎉🎉celebrate
Thanks Shalabh!!
s
Excellent!
p
@James Rosenthal I’ve been working on a gitlab/dagster cloud integration, and this is an example script (still in development) that might be useful to you: https://github.com/dagster-io/dagster-cloud-action/blob/main/gitlab/serverless-legacy-ci.yml
❤️ 1
That’s using the docker-based deploy. This is the script using the PEX-based deploy in a Gitlab action: https://github.com/dagster-io/dagster-cloud-action/blob/main/gitlab/serverless-ci.yml
Open in Slack
PreviousNext
Powered by