https://dagster.io/ logo
Join Slack
Powered by
I'm running into an issue with lightgbm on dagster...
# dagster-plus
k
I'm running into an issue with lightgbm on dagster cloud. I have had lightgbm in my requirements.txt, but I just now added code that called it (in a branch deployment). My code location didn't load, and returned "OSError: libgomp.so.1: cannot open shared object file: No such file or directory". It looks like the fix is easy if I had control over the dockerfile: https://stackoverflow.com/questions/55036740/lightgbm-inside-docker-libgomp-so-1-cannot-open-shared-object-file. I'm not sure what I can do myself though. How can I fix this?
Full stack trace when loading the code into my branch deployment:
Copy code
OSError: libgomp.so.1: cannot open shared object file: No such file or directory
  File "/usr/local/lib/python3.8/site-packages/dagster/_grpc/server.py", line 241, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
  File "/usr/local/lib/python3.8/site-packages/dagster/_grpc/server.py", line 104, in __init__
    loadable_targets = get_loadable_targets(
  File "/usr/local/lib/python3.8/site-packages/dagster/_grpc/utils.py", line 53, in get_loadable_targets
    else loadable_targets_from_python_package(package_name, working_directory)
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/workspace/autodiscovery.py", line 49, in loadable_targets_from_python_package
    module = load_python_module(
  File "/usr/local/lib/python3.8/site-packages/dagster/_core/code_pointer.py", line 138, in load_python_module
    return importlib.import_module(module_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/dagster/app/baker/__init__.py", line 63, in <module>
    from baker.nfl.repository import nfl as nfl_repository
  File "/opt/dagster/app/baker/nfl/repository.py", line 15, in <module>
    from baker.nfl import assets
  File "/opt/dagster/app/baker/nfl/assets/__init__.py", line 5, in <module>
    from baker.nfl.assets.simulation_assets import *
  File "/opt/dagster/app/baker/nfl/assets/simulation_assets.py", line 9, in <module>
    from baker.nfl.sim.nfl_simulation_runner import (
  File "/opt/dagster/app/baker/nfl/sim/nfl_simulation_runner.py", line 25, in <module>
    from baker.nfl.sim.prediction_models import PredictionModels
  File "/opt/dagster/app/baker/nfl/sim/prediction_models.py", line 4, in <module>
    from baker.nfl.ml.play_duration import PlayDuration
  File "/opt/dagster/app/baker/nfl/ml/play_duration.py", line 6, in <module>
    from lightgbm import LGBMRegressor
  File "/usr/local/lib/python3.8/site-packages/lightgbm/__init__.py", line 8, in <module>
    from .basic import Booster, Dataset, Sequence, register_logger
  File "/usr/local/lib/python3.8/site-packages/lightgbm/basic.py", line 110, in <module>
    _LIB = _load_lib()
  File "/usr/local/lib/python3.8/site-packages/lightgbm/basic.py", line 101, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/usr/local/lib/python3.8/ctypes/__init__.py", line 451, in LoadLibrary
    return self._dlltype(name)
  File "/usr/local/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
j
This is for a serverless or hybrid deployment? There’s a couple options for customizing the base image to include deps like this
k
Serverless
j
k
cool, thanks!
Got it. So I just need to build my own docker image and then tell dagster where to find it?
j
Correct, if you’re using the new python executable (pex) deploys
k
Okay. Sounds good. Thanks!
3 Views
Open in Slack
PreviousNext
Powered by