Web scraping

Links

• Easy web scraping with Scrapy (2019)

• A guide to Web Scraping without getting blocked in 2020

• Crawlab - Distributed web crawler admin platform for spiders management regardless of languages and frameworks.

• hakrawler - Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application.

• JobFunnel - Tool for scraping job websites, and filtering and reviewing the job listings.

• You-Get - Tiny command-line utility to download media contents (videos, audios, images) from the Web.

• Universal Reddit Scraper - Scrape Subreddits, Redditors, and comments on posts. A command-line tool written in Python.

• Gerapy - Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js.

• Ask HN: Best practices for ethical web scraping? (2020)

• Newscatcher - Programmatically collect normalized news from (almost) any website. (Code)

• scrapio - Simple and easy-to-use scraper and crawler in Go.

• Colly - Elegant Scraper and Crawler Framework for Golang.

• Python Web Scraping with Virtual Private Networks (2020)