Testing your spider script with Scrapy shell
As we previously see, it is kind of long to get the results of our Python script with Scrapy. You need to extract the result in a file in order to make it visible and it implies so many actions that you we are losing a lot of time.
By chance, there is a way to test your script without doing all that, this is what we call the Python shell. It simply consists in downloading a web page through the fetch command and then allow you to test all the different commands you want to perform. Let's see that with an example, first, we launch the shell:
scrapy shell
Congratulations, now you are within the shell. So, ask Scrapy which web page you would like to work on with the fetch command, for example:
fetch("https://twitter.com/yacy_search")
So, now the page has been reached so we can test our commands to see if they work:
response.xpath("//title/text()").extract()