TLDR Search

Enter a command. For example: od, git graft, az vm, docker image.

scrapy

Web-crawling framework. More information: https://scrapy.org.

• Create a project:
   
  scrapy startproject {{project_name}}
   
• Create a spider (in project directory):
   
  scrapy genspider {{spider_name}} {{website_domain}}
   
• Edit spider (in project directory):
   
  scrapy edit {{spider_name}}
   
• Run spider (in project directory):
   
  scrapy crawl {{spider_name}}
   
• Fetch a webpage as Scrapy sees it and print the source to stdout:
   
  scrapy fetch {{url}}
   
• Open a webpage in the default browser as Scrapy sees it (disable JavaScript for extra fidelity):
   
  scrapy view {{url}}
   
• Open Scrapy shell for URL, which allows interaction with the page source in a Python shell (or IPython if available):
   
  scrapy shell {{url}}