Hey! Sorry to keep asking, but I want an opinion on my use case: Currently I have multiple scrapy crawlers, each one for a different website. I have a solid factory that returns a solid per crawler, and then builds a pipeline with all the generated solids. The thing is, after the first crawler executes, the second one fails as the "reactor" that scrapy uses for its execution cannot be restarted. For this reason I opted to use the "Threads" package, and this solved the problem. The remaining problem is that the crawlers execute, but Dagit fails to detect that they are running. Dagster detects the startup period of the scrapers as their execution time (which is way lower than the normal execution time). What do you think?

Are you using it with asyncio or something?

Nope, no Asyncio, I simply use the multiprocess executor that dagster provides

No, I'm talking about Scrapy

Oh, sorry. I use the CrawlerProcess function. According to the docs, the function chooses that by itself (and its not specified in my settings).

def scrapy_solid(context):

    # The path seen from root (ie. from Repo.py) for "settings.py"
    settings_file_path = scrapy_proj_dir + "." + project + ".settings"

    # Temporary environment variable that sets the scrapy settings path
    os.environ.setdefault('SCRAPY_SETTINGS_MODULE', settings_file_path)

    # Project settings for the spider
    settings = get_project_settings()

    # Instantiate the crawler process with the project settings
    process = CrawlerProcess(settings)

    # Override the "FEED_URI" parameter for the scraper (name of the output file)
    crawler_class[i].custom_settings = {"FEED_URI": f"./{classNames[i]}_output.json"}

    # Configure and start crawler instance with a spider passed in
    process.crawl(crawler_class[i])
    Thread(target=process.start).start()

yield scrapy_solid

Yeah, you wouldn't to use that

What do you mean by concurrency model? The method it uses to do multi-processing?

Yes, how you are running your solids.

Thank you! I also wanted to ask you, what does the

join()

method do?

How familiar are you with Twisted?

Wooow, so this is waaay more complicated than I thought. Do you think it would be better to only use scrapy in this case?

No? This is a pretty normal thing you just need to understand how Scrapy runs internally.

Ok ok, I will look into it. Thank you so much @Noah K

we deploy the spiders on scrapinghub (now zyte IIRC) or a small ecr task -> then dump the data to s3. dagster picks it up from there

@Max Wong Awesome! So you deploy your spiders and Dagster then controls the execution of the spiders? How exactly do you call the deployed spiders?

nope. Dagster only picks up the file from spider runs. say, I know my spiders are done around 1AM. I then schedule a dagster task to start at 2AM to do whatever newer version of dagster has

sensors

, so it should come in quite handy

This was very useful. Thank you @Max Wong