Hey! I have been trying out sensors. But for some ...
# announcements
Hey! I have been trying out sensors. But for some reason the
errors out:
Copy code
2021-02-22 17:35:18 - SensorDaemon - INFO - Checking for new runs for the following sensors: some_sensor
2021-02-22 17:35:20 - SensorDaemon - ERROR - Sensor failed for some_sensor : dagster.core.errors.DagsterUserCodeProcessError: FileNotFoundError: [Errno 2] No such file or directory: 'sensor_repo.py'

Stack Trace:
  File "/usr/local/lib/python3.8/site-packages/dagster/grpc/server.py", line 356, in ListRepositories
  File "/usr/local/lib/python3.8/site-packages/dagster/grpc/server.py", line 107, in loadable_repository_symbols
  File "/usr/local/lib/python3.8/site-packages/dagster/grpc/server.py", line 97, in load
    self._loadable_repository_symbols = load_loadable_repository_symbols(
  File "/usr/local/lib/python3.8/site-packages/dagster/grpc/server.py", line 121, in load_loadable_repository_symbols
    loadable_targets = get_loadable_targets(
  File "/usr/local/lib/python3.8/site-packages/dagster/grpc/utils.py", line 25, in get_loadable_targets
    else loadable_targets_from_python_file(python_file, working_directory)
  File "/usr/local/lib/python3.8/site-packages/dagster/cli/workspace/autodiscovery.py", line 11, in loadable_targets_from_python_file
    loaded_module = load_python_file(python_file, working_directory)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/code_pointer.py", line 87, in load_python_file

Stack Trace:
  File "/usr/local/lib/python3.8/site-packages/dagster/scheduler/sensor.py", line 130, in execute_sensor_iteration
    with origin.create_handle() as repo_location_handle:
  File "/usr/local/lib/python3.8/site-packages/dagster/core/host_representation/origin.py", line 163, in create_handle
    return ManagedGrpcPythonEnvRepositoryLocationHandle(self)
  File "/usr/local/lib/python3.8/site-packages/dagster/core/host_representation/handle.py", line 223, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File "/usr/local/lib/python3.8/site-packages/dagster/api/list_repositories.py", line 16, in sync_list_repositories_grpc
    raise DagsterUserCodeProcessError(
The sensor repo is:
Copy code
@solid(config_schema={"filename": str})
def process_file(context):
    filename = context.solid_config["filename"]

def log_file_pipeline():

def some_sensor():
    yield RunRequest(
        run_config={"solids": {"process_file": {"config": {"filename": "hi"}}}},

def define_sensor_repository():
    repoDict = {}
    repoDict["sensors"] = {"some_sensor": some_sensor}
    repoDict["pipelines"] = {"log_file_pipeline": log_file_pipeline}
    return repoDict
Hi Jaikant - two questions - is the sensor_repo.py file available on whatever machine the dagster-daemon process is running on, and if so has the sensor_repo.py file been moved since you turned the sensor on?
@daniel I have the DAGSTER_HOME environment variable set to a directory different from where the sensor_repo.py file is located. The dagster-daemon is also started from a different directory from where the sensor_repo.py file is located. Could this be a problem?
Hm, it should use the path relative to your workspace.yaml file (assuming you're using a workspace.yaml file) but I will double-check - is it possible to share your workspace.yaml file?
(or alternatively, if you started the sensor from dagit or the CLI without using a workspace, what command you ran to do so)
I am not using a workspace.yaml. I am using the
cli commands to stop/start the sensor from the directory in which the sensor_repo.py file lies, using the
argument. Or dagit with the
argument. I can now, confirm the issue goes away if the
is started from the same directory in which the repo file lies …
Got it. -f does use the relative path (and store that relative path in a database for the daemon to later load the sensor code), so that makes sense. you'll need to either run dagster-daemon from the same folder, or use a workspace.yaml file, which will store it as an absolute path instead. This is something we should document better or change.
🙏 1
And filed https://github.com/dagster-io/dagster/issues/3723 to track fixing or better explaining this case, thanks for the report
thanks @daniel! 🙏
@daniel I am not sure if the issue is with the
relative path. But probably the dagster-daemon not utlising the
field in the jobs table. Tough issue as hard coding the paths takes away the flexibility 😬 - I am sure you guys have a solution for this in the worspace.yaml file which I still need to read up properly! I checked the db and the
table has a
field which is an absolute path and a location name which is a file name. The difficulty in the naming is symptomatic of the larger problem of flexibility probably!