Categorie: Python

How to use PyCharm to debug Scrapy projects

Create a Python file on crawler folder on your project. I used main.py.

  • Project
    • Crawler
      • Crawler
        • Spiders
        • …
      • main.py
      • scrapy.cfg

Inside your main.py put this code below.

from scrapy import cmdline    
cmdline.execute("scrapy crawl spider".split())

And you need to create a “Run Configuration” to run your main.py.

Doing this, if you put a breakpoint at your code it will stop there.

Excluding indexed directories in PyCharm

Since a while I am using the PyCharm Community Edition to expiriment with Python, but have been struggling with long startup times. This happens because of some Python package storing loads of httpcache on disk.

Solution

Here is how you can exclude these files and directories from the project that you are working on and speed up the startup process of Python:

  • Open Python;
  • Click File;
  • Click Settings;
  • Click on Project XYZ;
  • Select the folders to exclude.

You can download PyCharm (for Windows) here.