https://dagster.io/ logo
#ask-community
Title
# ask-community
g

Gowtham Manne

05/31/2022, 8:11 PM
Hi Everyone Can I know if we can start dagster and dagit, programmatically ?
s

Shriram Holla

05/31/2022, 8:15 PM
I’m not too sure from a dagster perspective but you could always have a bash script that does it and run that programmatically
g

Gowtham Manne

05/31/2022, 8:21 PM
when I run dagit or dagster , what happens internally, Can not I stimulate that programmatically to run the demon ?
s

Shriram Holla

05/31/2022, 8:27 PM
you can run jobs using
job.execute_in_process
. Not sure about dagit
g

Gowtham Manne

05/31/2022, 8:29 PM
currently I run dagit in the cmd, then using graphql Launching the jobs. But I don't want to run dagit as command in cli/cmd. I want to run dagit from python program.
s

Shriram Holla

05/31/2022, 8:33 PM
os.system('dagit ....')
?
g

Gowtham Manne

05/31/2022, 8:40 PM
should I point my script location in os path?
d

daniel

05/31/2022, 9:55 PM
Hi Gowtham - I'm a little confused what "programatically" means in this context. The way I would launch dagit programatically is to run the 'dagit' command in the command-line, but that's the usual way of running it so I think you are using the term programatically differently than me
The dagster command line can run within dagit running though - the step in the tutorial here shows how you can execute a job without any server running: https://docs.dagster.io/tutorial/intro-tutorial/single-op-job#dagster-cli
dagit is a webserver though - there isn't a way to run it other than as a server. If you want to run dagster jobs without a server running you'll need to use the dagster command line like in the link I posted
g

Gowtham Manne

06/01/2022, 1:21 PM
Hi @daniel I am trying to run dagit and demon , but not through CLI.
Copy code
from my_lib.dagster.daemon.cli import cli as dagster_daemon_cli
dagster_daemon_cli.main()


from my_lib.dagit import cli as dagit_cli
dagit_cli.main()
my_lib is the folder where I have all my python lib
d

daniel

06/01/2022, 1:22 PM
Are you trying to do both of them in the same process?
g

Gowtham Manne

06/01/2022, 1:27 PM
yes, should I do in two separate files ?
d

daniel

06/01/2022, 1:27 PM
You'll need them to be in two separate processes
since dagit is a webserver, it needs to run in its own process
similarly, the daemon is expecting to run in its own process
g

Gowtham Manne

06/01/2022, 1:31 PM
ModuleNotFoundError: No module named 'dagster._module_alias_map' when I ran daemon
ModuleNotFoundError: No module named 'dagster._check' when I ran dagit
d

daniel

06/01/2022, 1:34 PM
What version of the dagster python package do you have installed? Can you run "pip freeze | grep dagster" and "pip freeze | grep dagit" in the command line?
g

Gowtham Manne

06/01/2022, 1:37 PM
alembic==1.6.5 aniso8601==7.0.0 anyio==3.5.0 asgiref==3.5.0 attrs==21.4.0 bleach==5.0.0 certifi==2021.10.8 charset-normalizer==2.0.12 click==8.0.4 cloudpickle==2.0.0 colorama==0.4.4 coloredlogs==14.0 croniter==1.3.4 dagit==0.14.13 dagster==0.14.13 dagster-graphql==0.14.13 dask==2022.3.0 defusedxml==0.7.1 distributed==2022.3.0 docker==4.2.2 docstring-parser==0.14.1 entrypoints==0.4 fastapi==0.77.1 fsspec==2022.2.0 gql==2.0.0 graphene==2.1.9 graphql-core==2.3.2 graphql-relay==2.0.1 greenlet==1.1.2 grpcio==1.44.0 grpcio-health-checking==1.43.0 h11==0.13.0 HeapDict==1.0.1 httptools==0.4.0 humanfriendly==10.0 idna==3.3 importlib-resources==5.4.0 ipython-genutils==0.2.0 Jinja2==2.11.3 jsonschema==4.4.0 jupyter-core==4.10.0 locket==0.2.1 Mako==1.2.0 MarkupSafe==2.0.1 marshmallow==3.15.0 marshmallow-oneofschema==3.0.1 mistune==0.8.4 msgpack==1.0.3 mypy-extensions==0.4.3 nbconvert==5.6.1 nbformat==5.1.3 packaging==21.3 pandocfilters==1.5.0 partd==1.2.0 pendulum==2.1.2 pep562==1.1 prefect==1.1.0 promise==2.3 protobuf==3.20.1 psutil==5.9.0 pydantic==1.9.0 Pygments==2.12.0 pyparsing==3.0.7 pypiwin32==223 pyreadline3==3.4.1 pyrsistent==0.18.1 python-box==6.0.1 python-dateutil==2.8.2 python-dotenv==0.20.0 python-editor==1.0.4 python-slugify==6.1.1 pytz==2022.1 pytzdata==2020.1 pywin32==303 PyYAML==6.0 requests==2.27.1 Rx==1.6.1 six==1.16.0 sniffio==1.2.0 sortedcontainers==2.4.0 SQLAlchemy==1.4.36 starlette==0.19.1 tabulate==0.8.9 tblib==1.7.0 testpath==0.6.0 text-unidecode==1.3 toml==0.10.2 toolz==0.11.2 toposort==1.7 tornado==6.1 tqdm==4.64.0 traitlets==5.1.1 typing-compat==0.1.0 typing_extensions==4.2.0 urllib3==1.26.9 uvicorn==0.17.6 watchdog==2.1.7 watchgod==0.8.2 webencodings==0.5.1 websocket-client==1.3.1 websockets==10.3 zict==2.1.0 C:\Users\16155>pip freeze dagit alembic==1.6.5 aniso8601==7.0.0 anyio==3.5.0 asgiref==3.5.0 attrs==21.4.0 bleach==5.0.0 certifi==2021.10.8 charset-normalizer==2.0.12 click==8.0.4 cloudpickle==2.0.0 colorama==0.4.4 coloredlogs==14.0 croniter==1.3.4 dagit==0.14.13 dagster==0.14.13 dagster-graphql==0.14.13 dask==2022.3.0 defusedxml==0.7.1 distributed==2022.3.0 docker==4.2.2 docstring-parser==0.14.1 entrypoints==0.4 fastapi==0.77.1 fsspec==2022.2.0 gql==2.0.0 graphene==2.1.9 graphql-core==2.3.2 graphql-relay==2.0.1 greenlet==1.1.2 grpcio==1.44.0 grpcio-health-checking==1.43.0 h11==0.13.0 HeapDict==1.0.1 httptools==0.4.0 humanfriendly==10.0 idna==3.3 importlib-resources==5.4.0 ipython-genutils==0.2.0 Jinja2==2.11.3 jsonschema==4.4.0 jupyter-core==4.10.0 locket==0.2.1 Mako==1.2.0 MarkupSafe==2.0.1 marshmallow==3.15.0 marshmallow-oneofschema==3.0.1 mistune==0.8.4 msgpack==1.0.3 mypy-extensions==0.4.3 nbconvert==5.6.1 nbformat==5.1.3 packaging==21.3 pandocfilters==1.5.0 partd==1.2.0 pendulum==2.1.2 pep562==1.1 prefect==1.1.0 promise==2.3 protobuf==3.20.1 psutil==5.9.0 pydantic==1.9.0 Pygments==2.12.0 pyparsing==3.0.7 pypiwin32==223 pyreadline3==3.4.1 pyrsistent==0.18.1 python-box==6.0.1 python-dateutil==2.8.2 python-dotenv==0.20.0 python-editor==1.0.4 python-slugify==6.1.1 pytz==2022.1 pytzdata==2020.1 pywin32==303 PyYAML==6.0 requests==2.27.1 Rx==1.6.1 six==1.16.0 sniffio==1.2.0 sortedcontainers==2.4.0 SQLAlchemy==1.4.36 starlette==0.19.1 tabulate==0.8.9 tblib==1.7.0 testpath==0.6.0 text-unidecode==1.3 toml==0.10.2 toolz==0.11.2 toposort==1.7 tornado==6.1 tqdm==4.64.0 traitlets==5.1.1 typing-compat==0.1.0 typing_extensions==4.2.0 urllib3==1.26.9 uvicorn==0.17.6 watchdog==2.1.7 watchgod==0.8.2 webencodings==0.5.1 websocket-client==1.3.1 websockets==10.3 zict==2.1.0
d

daniel

06/01/2022, 1:37 PM
And you're getting the error that you posted earlier when you run "dagit" in the command-line?
The ModuleNotFoundError?
g

Gowtham Manne

06/01/2022, 1:39 PM
got above error when I ran
Copy code
from my_lib.dagster.daemon.cli import cli as dagster_daemon_cli
dagster_daemon_cli.main()
d

daniel

06/01/2022, 1:39 PM
What's my_lib?
g

Gowtham Manne

06/01/2022, 1:40 PM
pip install dagster, dagit -t my_lib
installed all python lib in a folder
d

daniel

06/01/2022, 1:42 PM
I don't think the dagster libraries are set up to be installed that way. My guess is the code in that folder is still pulling from the dagster packages that you have installed, rather than from the code in the my_lib folder
g

Gowtham Manne

06/01/2022, 1:44 PM
Can I know what this means ?DAGSTER_HOME=$PWD/dg python dagit.py
d

daniel

06/01/2022, 1:46 PM
Sorry, what is the "this" referring to in that question?
DAGSTER_HOME?
g

Gowtham Manne

06/01/2022, 1:47 PM
DAGSTER_HOME=$PWD/dg python dagit.py.
Yes sorry My bad
d

daniel

06/01/2022, 1:47 PM
g

Gowtham Manne

06/01/2022, 1:48 PM
I removed my_lib and ran. It started throwing different error
dagster.core.errors.DagsterHomeNotSetError: The environment variable $DAGSTER_HOME is not set. Dagster requires this environment variable to be set to an existing directory in your filesystem. This directory is used to store metadata across sessions, or load the dagster.yaml file which can configure storing metadata in an external database. You can resolve this error by exporting the environment variable. For example, you can run the following command in your shell or include it in your shell configuration file: export DAGSTER_HOME=~"/dagster_home" or PowerShell $env:DAGSTER_HOME = ($home + '\dagster_home')or batchset DAGSTER_HOME=%UserProfile%/dagster_homeAlternatively, DagsterInstance.ephemeral() can be used for a transient instance.
d

daniel

06/01/2022, 1:48 PM
I think that error message is a pretty good summary actually - do you have any specific questions about it?
g

Gowtham Manne

06/01/2022, 1:57 PM
was able to run dagit, but trying to set DAGSTER_HOME
Copy code
from dagster.daemon.cli import cli as dagster_daemon_cli
import sys,os


#making a dagster home dir
dagster_home = "dg"
path = os.path.join(os.getcwd(), dagster_home)
if not os.path.exists(path):
    os.mkdir(path)

#setting dagster home
print(os.getcwd()+'\\'+dagster_home)
os.environ["DAGSTER_HOME"] = os.getcwd() +'/'+dagster_home

#Running dagster daemon
sys.argv = [sys.argv[0], "run", "--empty-workspace"]
dagster_daemon_cli.main()
This is what my code looks like
logs looks like below
2022-06-01 095044 -0500 - dagster.daemon - INFO - instance is configured with the following daemons: ['BackfillDaemon', 'SchedulerDaemon', 'SensorDaemon'] 2022-06-01 095044 -0500 - dagster.daemon.SensorDaemon - INFO - Not checking for any runs since no sensors have been started. 2022-06-01 095144 -0500 - dagster.daemon.SensorDaemon - INFO - Not checking for any runs since no sensors have been started.
I hope dagster daemon is started now
d

daniel

06/01/2022, 2:52 PM
that looks right to me
1
g

Gowtham Manne

06/01/2022, 2:53 PM
Thanks for your help!
Sorry One last question, Is there a way I can supply port number in above code ?
d

daniel

06/01/2022, 5:31 PM
The daemon isn't a server, so there isn't a port to override
it launches jobs based on reading from the database, it doesn't take requests from clients
228 Views