https://dagster.io/ logo
Join Slack
Powered by
Hey Alex - when you run dagit you need to tell it ...
# ask-community
d
Hey Alex - when you run dagit you need to tell it where to load your code from. There are some example arguments to dagit here to do this: https://docs.dagster.io/concepts/code-locations#deploying-and-loading-code-locations
o
but i have described module.name
d
oh, it's possible you need to upgrade your dagster version - the pyproject.toml stuff was added in 1.1.7
the error message looks like it's from an earlier version than that
o
ahh... Okay, i`ll try
d
Copy code
ModuleNotFoundError: No module named 'meltano.core'
is the key line there I think - do you have that package installed?
o
I thought so too, but I took the dependencies from the setup.py of the source repository
d
which means it'll only install that if you add a [meltano] to the pip install
o
Looks like it has a meltano extra
I also added this one
I have meltano installed globally
d
What happens if you run "python -m saasalerts_dagster"? do you get the same error?
o
i see that i have installed dagster
d
It seems like you're running into some general python environment / import issues - are you using virtual environments? I find those pretty useful to avoid these classes of import issues: https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment
o
yes i use venv and i make it by this guide
d
Got it - what I can say is that that earlier error seems to believe pretty strongly that meltano isn't actually installed - i would try installing it in the particular environment that dagster is using rather than relying on it being globally available
o
But the error says that it does not find the dagster package...
d
I suspect that second error has to do with the specific environment that the "python" executable is pointing at
likely unrelated to the meltano issue, other than indicative that there are some confusing things related to python environments happening on your machine
o
So you're suggesting that you just run pip install meltano inside a virtual env?
d
I don't have quite enough of an understanding about your environment to have a specific recommendation yet - but I can say that it's pretty confident that the environment where your dagster code is running isn't able to find the meltano package
o
I don't understand how to avoid this. It seems to me that it is correct that meltano is installed globally
d
Generally when using venvs I try to rely only on packages installed in the venv - so the first thing I would try would be including meltano explicitly in the set of packages that are installed in that venv
New errors means progress 🙂 now it seems like dagster_meltano isn't installed
o
yes, i see, but i already run pip install dagster_meltano
d
And it succeeded and was in the same venv that dagster is using?
o
yeah
d
Are you certain it installed locally and not from pypi? I see a dagster-meltano here: https://pypi.org/project/dagster-meltano/#history
but it looks like you wanted to install it editably from that github repo
or locally from your own project, rather
o
It seems not ... I do not understand where it should be. But what I know for sure is that the gitlab project is up and running (without additional pip install)
and me repo is pretty copy of this
d
To install something locally I would typically do something like 'pip install -e <path to my local dagster-meltano folder>'
o
look like i resolved all of problems with packages
I don`t understand why it say thay i do not have any assets
d
Is it possible to share the code in the module that you pointed dagster at in your pyproject.toml
o
thanks a lot for your time and help
d
How about the code of the module itself?
o
which? On screenshot you can see my file tree
d
How about the contents of the init file
You'll want it in your init.py file
o
it is noteworthy that I just copied this part from the original repository
d
I think that Github repository was intended to be used as a library that's imported from some other package
o
Hmmm, it seems not. I see that there is a meltano project inside that I want to edit (change the yaml file)
Even so .. I think that if you clone this repository, then it will be possible to run it locally without any problems
@shailesh What you think about this?
I think that Github repository was intended to be used as a library that's imported from some other package
d
I can't provide in-depth support for that github repository since we didn't write it - but glancing through it, it doesn't look like it includes a Dagster repository or Definitions file, which are needed to run dagster code - the docs link I posted most recently provides instructions on how to set that up
o
Hmm, but there is a setup.py that installs all these libraries
d
From briefly looking over the code it looks to me like dagster-meltano is expected to be a package that you import from your own code that includes some utility assets/ops/etc., like dagster-kubernetes or dagster-airbyte
o
Then I don't understand how to use it. So how do I run some meltano command or add some meltano tap/target?
d
I'm not familiar enough with meltano or that specific github repo to provide specifics, but I think broadly it would involve importing the utility functions from dagster-meltano and calling them within your own code - so for example I see a load_jobs_from_meltano_project function, so in your own dagster code you could potentially do something like this in your init file:
Copy code
from dagster_meltano import load_jobs_from_meltano_project
from dagster import Definitions

defs = Definitions(jobs=load_jobs_from_meltano_project(YOUR_PROJECT_DIR_HERE))
o
For me, it looked like code that I can use in my project, where I can change/customize the existing description of op/jobs/assets and add them to the dependencies. And the main one, where i can modify meltano project
d
Jules can say for sure - I do see an example Dagster repository here in the repo that demonstrates how one might use those utility functions, which is pretty in line with what I described just above: https://github.com/quantile-development/dagster-meltano/blob/development/meltano/orchestrate/repository.py
importing that repository (or a Definitions class) in your init file should make dagit able to find some actual jobs
2 Views
Open in Slack
PreviousNext
Powered by