Keval
11/18/2020, 7:29 PMdownload_file("file_path")
sandy
11/19/2020, 12:27 AM<https://docs.dagster.io/_apidocs/execution#dagster.execute_solid>
, which accepts a dynamic value. Do you need to execute a full pipeline on dynamic inputs? This is a little bit more involvedKeval
11/19/2020, 8:58 PMsandy
11/20/2020, 7:37 PMalex
11/20/2020, 9:25 PMsolids:
download_file:
inputs:
link: <the link>
Keval
11/29/2020, 10:24 AMdagster.core.errors.DagsterUserCodeProcessError: (DagsterInvalidDefinitionError) - dagster.core.errors.DagsterInvalidDefinitionError: @solid 'get_url' decorated function does not have required positional parameter 'context'. Solid functions should only have keyword arguments that match input names and a first positional parameter named 'context'.
Stack Trace:
File "c:\users\kotha\appdata\local\programs\python\python38-32\lib\site-packages\dagster\grpc\server.py", line 336, in ListRepositories
self._repository_symbols_and_code_pointers.loadable_repository_symbols,
File "c:\users\kotha\appdata\local\programs\python\python38-32\lib\site-packages\dagster\grpc\server.py", line 106, in loadable_repository_symbols
self.load()
File "c:\users\kotha\appdata\local\programs\python\python38-32\lib\site-packages\dagster\grpc\server.py", line 96, in load
self._loadable_repository_symbols = load_loadable_repository_symbols(
File "c:\users\kotha\appdata\local\programs\python\python38-32\lib\site-packages\dagster\grpc\server.py", line 120, in load_loadable_repository_symbols
loadable_targets = get_loadable_targets(
File "c:\users\kotha\appdata\local\programs\python\python38-32\lib\site-packages\dagster\grpc\utils.py", line 25, in get_loadable_targets
else loadable_targets_from_python_file(python_file, working_directory)
File "c:\users\kotha\appdata\local\programs\python\python38-32\lib\site-packages\dagster\cli\workspace\autodiscovery.py", line 11, in loadable_targets_from_python_file
loaded_module = load_python_file(python_file, working_directory)
File "c:\users\kotha\appdata\local\programs\python\python38-32\lib\site-packages\dagster\core\code_pointer.py", line 95, in load_python_file
return import_module_from_path(module_name, python_file)
File "c:\users\kotha\appdata\local\programs\python\python38-32\lib\site-packages\dagster\seven\__init__.py", line 118, in import_module_from_path
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "hello_dagster.py", line 4, in <module>
import Utils
File "C:\Users\kotha\Dropbox\Sample Data Sets\datavalidation\Dagster\Utils.py", line 12, in <module>
def get_url(url):
File "c:\users\kotha\appdata\local\programs\python\python38-32\lib\site-packages\dagster\core\definitions\decorators\solid.py", line 196, in solid
return _Solid()(name)
File "c:\users\kotha\appdata\local\programs\python\python38-32\lib\site-packages\dagster\core\definitions\decorators\solid.py", line 67, in __call__
positional_inputs = validate_solid_fn("@solid", self.name, fn, input_defs, ["context"])
File "c:\users\kotha\appdata\local\programs\python\python38-32\lib\site-packages\dagster\core\definitions\decorators\solid.py", line 311, in validate_solid_fn
raise DagsterInvalidDefinitionError(
@solid
def get_url(url):
#return '<https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv>'
return url
@solid
def download_file(link):
print('Downloading File...')
#<http://context.log.info|context.log.info>("Downloading File From URL:"+link)
req = requests.get(link,stream=True)
url_content = req.content
file = link.split('/')[-1]
data_file = open(file, 'wb')
data_file.write(url_content)
data_file.close()
return file
hello_dagster.py
@pipeline
def hello_pipeline():
Utils.download_file(Utils.get_url('<https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv')>)
Can some one help what exactly wrong here ?alex
11/30/2020, 3:36 PM@solid ‘get_url’ decorated function does not have required positional parameter ‘context’. Solid functions should only have keyword arguments that match input names and a first positional parameter named ‘context’.the first parameter to your
@solid
functions need to be context
followed by ones you want for parameters. You can use _
instead if you don’t need to use anything on the context