https://dagster.io/ logo
Join Slack
Powered by
Hi Everyone Can I know if we can start dagster and...
# ask-community
g
Hi Everyone Can I know if we can start dagster and dagit, programmatically ?
s
I’m not too sure from a dagster perspective but you could always have a bash script that does it and run that programmatically
g
when I run dagit or dagster , what happens internally, Can not I stimulate that programmatically to run the demon ?
s
you can run jobs using 
job.execute_in_process
. Not sure about dagit
g
currently I run dagit in the cmd, then using graphql Launching the jobs. But I don't want to run dagit as command in cli/cmd. I want to run dagit from python program.
s
os.system('dagit ....')
?
g
should I point my script location in os path?
d
Hi Gowtham - I'm a little confused what "programatically" means in this context. The way I would launch dagit programatically is to run the 'dagit' command in the command-line, but that's the usual way of running it so I think you are using the term programatically differently than me
The dagster command line can run within dagit running though - the step in the tutorial here shows how you can execute a job without any server running: https://docs.dagster.io/tutorial/intro-tutorial/single-op-job#dagster-cli
dagit is a webserver though - there isn't a way to run it other than as a server. If you want to run dagster jobs without a server running you'll need to use the dagster command line like in the link I posted
g
Hi @daniel I am trying to run dagit and demon , but not through CLI.
Copy code
from my_lib.dagster.daemon.cli import cli as dagster_daemon_cli
dagster_daemon_cli.main()


from my_lib.dagit import cli as dagit_cli
dagit_cli.main()
my_lib is the folder where I have all my python lib
d
Are you trying to do both of them in the same process?
g
yes, should I do in two separate files ?
d
You'll need them to be in two separate processes
since dagit is a webserver, it needs to run in its own process
similarly, the daemon is expecting to run in its own process
g
ModuleNotFoundError: No module named 'dagster._module_alias_map' when I ran daemon
ModuleNotFoundError: No module named 'dagster._check' when I ran dagit
d
What version of the dagster python package do you have installed? Can you run "pip freeze | grep dagster" and "pip freeze | grep dagit" in the command line?
g
alembic==1.6.5 aniso8601==7.0.0 anyio==3.5.0 asgiref==3.5.0 attrs==21.4.0 bleach==5.0.0 certifi==2021.10.8 charset-normalizer==2.0.12 click==8.0.4 cloudpickle==2.0.0 colorama==0.4.4 coloredlogs==14.0 croniter==1.3.4 dagit==0.14.13 dagster==0.14.13 dagster-graphql==0.14.13 dask==2022.3.0 defusedxml==0.7.1 distributed==2022.3.0 docker==4.2.2 docstring-parser==0.14.1 entrypoints==0.4 fastapi==0.77.1 fsspec==2022.2.0 gql==2.0.0 graphene==2.1.9 graphql-core==2.3.2 graphql-relay==2.0.1 greenlet==1.1.2 grpcio==1.44.0 grpcio-health-checking==1.43.0 h11==0.13.0 HeapDict==1.0.1 httptools==0.4.0 humanfriendly==10.0 idna==3.3 importlib-resources==5.4.0 ipython-genutils==0.2.0 Jinja2==2.11.3 jsonschema==4.4.0 jupyter-core==4.10.0 locket==0.2.1 Mako==1.2.0 MarkupSafe==2.0.1 marshmallow==3.15.0 marshmallow-oneofschema==3.0.1 mistune==0.8.4 msgpack==1.0.3 mypy-extensions==0.4.3 nbconvert==5.6.1 nbformat==5.1.3 packaging==21.3 pandocfilters==1.5.0 partd==1.2.0 pendulum==2.1.2 pep562==1.1 prefect==1.1.0 promise==2.3 protobuf==3.20.1 psutil==5.9.0 pydantic==1.9.0 Pygments==2.12.0 pyparsing==3.0.7 pypiwin32==223 pyreadline3==3.4.1 pyrsistent==0.18.1 python-box==6.0.1 python-dateutil==2.8.2 python-dotenv==0.20.0 python-editor==1.0.4 python-slugify==6.1.1 pytz==2022.1 pytzdata==2020.1 pywin32==303 PyYAML==6.0 requests==2.27.1 Rx==1.6.1 six==1.16.0 sniffio==1.2.0 sortedcontainers==2.4.0 SQLAlchemy==1.4.36 starlette==0.19.1 tabulate==0.8.9 tblib==1.7.0 testpath==0.6.0 text-unidecode==1.3 toml==0.10.2 toolz==0.11.2 toposort==1.7 tornado==6.1 tqdm==4.64.0 traitlets==5.1.1 typing-compat==0.1.0 typing_extensions==4.2.0 urllib3==1.26.9 uvicorn==0.17.6 watchdog==2.1.7 watchgod==0.8.2 webencodings==0.5.1 websocket-client==1.3.1 websockets==10.3 zict==2.1.0 C:\Users\16155>pip freeze dagit alembic==1.6.5 aniso8601==7.0.0 anyio==3.5.0 asgiref==3.5.0 attrs==21.4.0 bleach==5.0.0 certifi==2021.10.8 charset-normalizer==2.0.12 click==8.0.4 cloudpickle==2.0.0 colorama==0.4.4 coloredlogs==14.0 croniter==1.3.4 dagit==0.14.13 dagster==0.14.13 dagster-graphql==0.14.13 dask==2022.3.0 defusedxml==0.7.1 distributed==2022.3.0 docker==4.2.2 docstring-parser==0.14.1 entrypoints==0.4 fastapi==0.77.1 fsspec==2022.2.0 gql==2.0.0 graphene==2.1.9 graphql-core==2.3.2 graphql-relay==2.0.1 greenlet==1.1.2 grpcio==1.44.0 grpcio-health-checking==1.43.0 h11==0.13.0 HeapDict==1.0.1 httptools==0.4.0 humanfriendly==10.0 idna==3.3 importlib-resources==5.4.0 ipython-genutils==0.2.0 Jinja2==2.11.3 jsonschema==4.4.0 jupyter-core==4.10.0 locket==0.2.1 Mako==1.2.0 MarkupSafe==2.0.1 marshmallow==3.15.0 marshmallow-oneofschema==3.0.1 mistune==0.8.4 msgpack==1.0.3 mypy-extensions==0.4.3 nbconvert==5.6.1 nbformat==5.1.3 packaging==21.3 pandocfilters==1.5.0 partd==1.2.0 pendulum==2.1.2 pep562==1.1 prefect==1.1.0 promise==2.3 protobuf==3.20.1 psutil==5.9.0 pydantic==1.9.0 Pygments==2.12.0 pyparsing==3.0.7 pypiwin32==223 pyreadline3==3.4.1 pyrsistent==0.18.1 python-box==6.0.1 python-dateutil==2.8.2 python-dotenv==0.20.0 python-editor==1.0.4 python-slugify==6.1.1 pytz==2022.1 pytzdata==2020.1 pywin32==303 PyYAML==6.0 requests==2.27.1 Rx==1.6.1 six==1.16.0 sniffio==1.2.0 sortedcontainers==2.4.0 SQLAlchemy==1.4.36 starlette==0.19.1 tabulate==0.8.9 tblib==1.7.0 testpath==0.6.0 text-unidecode==1.3 toml==0.10.2 toolz==0.11.2 toposort==1.7 tornado==6.1 tqdm==4.64.0 traitlets==5.1.1 typing-compat==0.1.0 typing_extensions==4.2.0 urllib3==1.26.9 uvicorn==0.17.6 watchdog==2.1.7 watchgod==0.8.2 webencodings==0.5.1 websocket-client==1.3.1 websockets==10.3 zict==2.1.0
d
And you're getting the error that you posted earlier when you run "dagit" in the command-line?
The ModuleNotFoundError?
g
got above error when I ran
Copy code
from my_lib.dagster.daemon.cli import cli as dagster_daemon_cli
dagster_daemon_cli.main()
d
What's my_lib?
g
pip install dagster, dagit -t my_lib
installed all python lib in a folder
d
I don't think the dagster libraries are set up to be installed that way. My guess is the code in that folder is still pulling from the dagster packages that you have installed, rather than from the code in the my_lib folder
g
Can I know what this means ?DAGSTER_HOME=$PWD/dg python dagit.py
d
Sorry, what is the "this" referring to in that question?
DAGSTER_HOME?
g
DAGSTER_HOME=$PWD/dg python dagit.py.
Yes sorry My bad
d
g
I removed my_lib and ran. It started throwing different error
dagster.core.errors.DagsterHomeNotSetError: The environment variable $DAGSTER_HOME is not set. Dagster requires this environment variable to be set to an existing directory in your filesystem. This directory is used to store metadata across sessions, or load the dagster.yaml file which can configure storing metadata in an external database. You can resolve this error by exporting the environment variable. For example, you can run the following command in your shell or include it in your shell configuration file: export DAGSTER_HOME=~"/dagster_home" or PowerShell $env:DAGSTER_HOME = ($home + '\dagster_home')or batchset DAGSTER_HOME=%UserProfile%/dagster_homeAlternatively, DagsterInstance.ephemeral() can be used for a transient instance.
d
I think that error message is a pretty good summary actually - do you have any specific questions about it?
g
was able to run dagit, but trying to set DAGSTER_HOME
Copy code
from dagster.daemon.cli import cli as dagster_daemon_cli
import sys,os


#making a dagster home dir
dagster_home = "dg"
path = os.path.join(os.getcwd(), dagster_home)
if not os.path.exists(path):
    os.mkdir(path)

#setting dagster home
print(os.getcwd()+'\\'+dagster_home)
os.environ["DAGSTER_HOME"] = os.getcwd() +'/'+dagster_home

#Running dagster daemon
sys.argv = [sys.argv[0], "run", "--empty-workspace"]
dagster_daemon_cli.main()
This is what my code looks like
logs looks like below
2022-06-01 095044 -0500 - dagster.daemon - INFO - instance is configured with the following daemons: ['BackfillDaemon', 'SchedulerDaemon', 'SensorDaemon'] 2022-06-01 095044 -0500 - dagster.daemon.SensorDaemon - INFO - Not checking for any runs since no sensors have been started. 2022-06-01 095144 -0500 - dagster.daemon.SensorDaemon - INFO - Not checking for any runs since no sensors have been started.
I hope dagster daemon is started now
d
that looks right to me
✅ 1
g
Thanks for your help!
Sorry One last question, Is there a way I can supply port number in above code ?
d
The daemon isn't a server, so there isn't a port to override
it launches jobs based on reading from the database, it doesn't take requests from clients
299 Views
Open in Slack
PreviousNext
Powered by