19913d85-f662-4518-a9a1-7adc1793dfd4.png){kind=small}
Channels
#dagster-support
Title
# dagster-support
d
daniel
12/27/2022, 3:28 PMHey Alex - when you run dagit you need to tell it where to load your code from. There are some example arguments to dagit here to do this: https://docs.dagster.io/concepts/code-locations#deploying-and-loading-code-locations
o
Oleksandr (Alex) Dimov
12/27/2022, 3:30 PMbut i have described module.name
d
daniel
12/27/2022, 3:31 PMoh, it's possible you need to upgrade your dagster version - the pyproject.toml stuff was added in 1.1.7
the error message looks like it's from an earlier version than that
o
Oleksandr (Alex) Dimov
12/27/2022, 3:33 PMahh... Okay, i`ll try
d
daniel
12/27/2022, 3:40 PM Copy code
ModuleNotFoundError: No module named 'meltano.core'o
Oleksandr (Alex) Dimov
12/27/2022, 3:41 PMI thought so too, but I took the dependencies from the setup.py of the source repository
d
daniel
12/27/2022, 3:42 PMLooks like it has a meltano extra: https://github.com/quantile-development/dagster-meltano/blob/development/setup.py#L32-L34
which means it'll only install that if you add a [meltano] to the pip install
o
Oleksandr (Alex) Dimov
12/27/2022, 3:43 PMLooks like it has a meltano extraI also added this one
I have meltano installed globally
d
daniel
12/27/2022, 3:44 PMWhat happens if you run "python -m saasalerts_dagster"? do you get the same error?
o
Oleksandr (Alex) Dimov
12/27/2022, 3:47 PMi see that i have installed dagster
d
daniel
12/27/2022, 3:47 PMIt seems like you're running into some general python environment / import issues - are you using virtual environments? I find those pretty useful to avoid these classes of import issues: https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment
o
Oleksandr (Alex) Dimov
12/27/2022, 3:49 PMyes i use venv and i make it by this guide
d
daniel
12/27/2022, 3:50 PMGot it - what I can say is that that earlier error seems to believe pretty strongly that meltano isn't actually installed - i would try installing it in the particular environment that dagster is using rather than relying on it being globally available
o
Oleksandr (Alex) Dimov
12/27/2022, 3:51 PMBut the error says that it does not find the dagster package...
d
daniel
12/27/2022, 3:51 PMI suspect that second error has to do with the specific environment that the "python" executable is pointing at
likely unrelated to the meltano issue, other than indicative that there are some confusing things related to python environments happening on your machine
o
Oleksandr (Alex) Dimov
12/27/2022, 3:52 PMSo you're suggesting that you just run pip install meltano inside a virtual env?
d
daniel
12/27/2022, 3:53 PMI don't have quite enough of an understanding about your environment to have a specific recommendation yet - but I can say that it's pretty confident that the environment where your dagster code is running isn't able to find the meltano package
o
Oleksandr (Alex) Dimov
12/27/2022, 3:53 PMI don't understand how to avoid this. It seems to me that it is correct that meltano is installed globally
d
daniel
12/27/2022, 3:54 PMGenerally when using venvs I try to rely only on packages installed in the venv - so the first thing I would try would be including meltano explicitly in the set of packages that are installed in that venv
New errors means progress 🙂 now it seems like dagster_meltano isn't installed
o
Oleksandr (Alex) Dimov
12/27/2022, 3:58 PMyes, i see, but i already run pip install dagster_meltano
d
daniel
12/27/2022, 3:59 PMAnd it succeeded and was in the same venv that dagster is using?
o
Oleksandr (Alex) Dimov
12/27/2022, 3:59 PMyeah
d
daniel
12/27/2022, 4:00 PMAre you certain it installed locally and not from pypi? I see a dagster-meltano here: https://pypi.org/project/dagster-meltano/#history
but it looks like you wanted to install it editably from that github repo
or locally from your own project, rather
o
Oleksandr (Alex) Dimov
12/27/2022, 4:01 PMIt seems not ... I do not understand where it should be. But what I know for sure is that the gitlab project is up and running (without additional pip install)
and me repo is pretty copy of this
d
daniel
12/27/2022, 4:02 PMTo install something locally I would typically do something like 'pip install -e <path to my local dagster-meltano folder>'
o
Oleksandr (Alex) Dimov
12/27/2022, 4:07 PMlook like i resolved all of problems with packages
I don`t understand why it say thay i do not have any assets
d
daniel
12/27/2022, 4:08 PMIs it possible to share the code in the module that you pointed dagster at in your pyproject.toml
o
Oleksandr (Alex) Dimov
12/27/2022, 4:09 PMthanks a lot for your time and help
d
daniel
12/27/2022, 4:10 PMHow about the code of the module itself?
o
Oleksandr (Alex) Dimov
12/27/2022, 4:10 PMwhich? On screenshot you can see my file tree
d
daniel
12/27/2022, 4:11 PMHow about the contents of the init file
Do you have a Definitions object defined? https://docs.dagster.io/concepts/code-locations#defining-code-locations
You'll want it in your init.py file
o
Oleksandr (Alex) Dimov
12/27/2022, 4:14 PMit is noteworthy that I just copied this part from the original repository
d
daniel
12/27/2022, 4:15 PMI think that Github repository was intended to be used as a library that's imported from some other package
o
Oleksandr (Alex) Dimov
12/27/2022, 4:16 PMHmmm, it seems not. I see that there is a meltano project inside that I want to edit (change the yaml file)
Even so .. I think that if you clone this repository, then it will be possible to run it locally without any problems
@shailesh What you think about this?
I think that Github repository was intended to be used as a library that's imported from some other package
d
daniel
12/27/2022, 4:21 PMI can't provide in-depth support for that github repository since we didn't write it - but glancing through it, it doesn't look like it includes a Dagster repository or Definitions file, which are needed to run dagster code - the docs link I posted most recently provides instructions on how to set that up
o
Oleksandr (Alex) Dimov
12/27/2022, 4:22 PMHmm, but there is a setup.py that installs all these libraries
d
daniel
12/27/2022, 4:24 PMFrom briefly looking over the code it looks to me like dagster-meltano is expected to be a package that you import from your own code that includes some utility assets/ops/etc., like dagster-kubernetes or dagster-airbyte
o
Oleksandr (Alex) Dimov
12/27/2022, 4:26 PMThen I don't understand how to use it. So how do I run some meltano command or add some meltano tap/target?
d
daniel
12/27/2022, 4:34 PMI'm not familiar enough with meltano or that specific github repo to provide specifics, but I think broadly it would involve importing the utility functions from dagster-meltano and calling them within your own code - so for example I see a load_jobs_from_meltano_project function, so in your own dagster code you could potentially do something like this in your init file:
Copy code
from dagster_meltano import load_jobs_from_meltano_project
from dagster import Definitions
defs = Definitions(jobs=load_jobs_from_meltano_project(YOUR_PROJECT_DIR_HERE))o
Oleksandr (Alex) Dimov
12/27/2022, 4:36 PMFor me, it looked like code that I can use in my project, where I can change/customize the existing description of op/jobs/assets and add them to the dependencies.
And the main one, where i can modify meltano project
d
daniel
12/27/2022, 4:39 PMJules can say for sure - I do see an example Dagster repository here in the repo that demonstrates how one might use those utility functions, which is pretty in line with what I described just above: https://github.com/quantile-development/dagster-meltano/blob/development/meltano/orchestrate/repository.py
importing that repository (or a Definitions class) in your init file should make dagit able to find some actual jobs
2 Views