Woodrow.wiki
  • Hello World
  • Meta
    • Worflow & Structure
  • Footprint
    • Cover Letter
  • Cheatsheets
    • Awesome-Cheatsheets
  • Writing
    • Markdown
  • Art
    • Tattoos
    • Music
  • Computer Science
    • Data Structures
    • Algorithms
      • Compression
    • Parsing
    • Formal verification
      • TLA+
    • Automata theory
  • Programming
    • Git
    • Concurrency
    • Regex
    • Hashing
    • Functional programming
    • Semantic versioning
    • Reverse engineering
    • Protocol Buffers
    • Coding practice
    • Serialization
    • Competitive Programming
    • Design patterns
    • System Design
    • Continuous Integration
    • Documentation
    • Embedded systems
    • Encoding
    • Interactive computing
    • Quantum programming
  • Web
    • Browsers
      • Safari
      • Google Chrome
        • Chrome DevTools
      • Firefox
      • Bookmarklets
      • Stylish themes
        • Web performance
        • Service workers
        • Static sites
      • Hugo
        • Node.js
        • Deno
        • WebAssembly
        • SEO
        • Electron
        • Webpack
        • Rollup
        • WebRTC
        • Search engines
        • Web engines
      • WebKit
        • Progressive Web Apps
        • Web workers
        • Web scraping
    • RSS
    • Web accessibility
      • Content management systems
      • JAMstack
        • Redwood
  • Front End
    • CSS
      • CSS Grid
      • Tailwind CSS
      • CSS in JS
    • HTML
  • Programming languages
    • Go
      • Go libraries
    • Python
      • Python libraries
    • Swift
      • Swift libraries
    • Rust
      • Rust libraries
    • Haskell
      • Haskell libraries
    • JavaScript
      • ESLint
      • Babel
    • JS libraries
      • Gatsby JS
      • Next.js
      • React
      • Three.js
      • Expo
      • Vue.js
    • TypeScript
      • TypeScript libraries
    • Scala
      • Scala libraries
    • Bash
    • Clojure
      • ClojureScript
    • Erlang
    • Java
      • Java libraries
    • Kotlin
      • Kotlin libraries
    • C
      • C libraries
    • Lua
    • Ruby
    • C++
      • C++ libraries
    • Julia
      • Julia libraries
    • Dart
      • Flutter
    • R
    • Assembly
    • Processing
  • Data Science
    • Visualization
  • Open Source
  • Text editors
    • Vim
      • Vim plugins
    • Sublime Text
      • Sublime Text plugins
    • Emacs
  • Distributed systems
    • RPCs
      • gRPC
    • Load balancing
    • Message queue
      • ZeroMQ
  • Containers
    • Kubernetes
    • Docker
  • Cloud computing
    • AWS
      • AWS Lambda
  • Operating systems
    • Linux
      • NixOS
    • BSD
    • Windows
  • Package managers
    • brew
  • DevOps
    • SRE
      • Security
  • Unix
    • Shell
      • Zsh
        • Zsh plugins
    • My file system
    • Dotfiles
  • Networking
    • HTTP
    • DNS
    • Peer to peer
      • IPFS
      • BitTorrent
    • Internet of things
      • LoRaWAN
    • Microservices
    • Decentralization
    • Nginx
    • VPN
      • WireGuard
    • GraphQL
    • SSH
    • Domains
    • QUIC
    • WebSocket
    • File sharing
  • Tools
    • CodeSandbox
    • GitBook
    • Dropbox
    • Telegram
    • Product Hunt
    • DuckDuckGo
    • IFTTT
    • Dat
    • Email
  • Analytics
  • Databases
    • PostgreSQL
    • Redis
    • RabbitMQ
    • DynamoDB
    • FoundationDB
    • SQL
    • Memcached
  • Psychology
    • Addiction
    • Biases
    • Negotiating
    • Marketing
  • Work
    • Finding work
      • Interviews
      • CV
      • Hiring
      • Contracting
    • Remote work
    • Consultancies
  • Management
    • Product Management
  • CLI
    • sed
    • tmux
  • Podcasts
Powered by GitBook
On this page

Was this helpful?

  1. Web
  2. Browsers
  3. WebKit

Web scraping

PreviousWeb workersNextRSS

Last updated 5 years ago

Was this helpful?

Links

  • - Distributed web crawler admin platform for spiders management regardless of languages and frameworks.

  • - Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application.

  • - Tool for scraping job websites, and filtering and reviewing the job listings.

  • - Tiny command-line utility to download media contents (videos, audios, images) from the Web.

  • - Scrape Subreddits, Redditors, and comments on posts. A command-line tool written in Python.

  • - Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js.

  • - Programmatically collect normalized news from (almost) any website. ()

  • - Simple and easy-to-use scraper and crawler in Go.

  • - Elegant Scraper and Crawler Framework for Golang.

Easy web scraping with Scrapy (2019)
A guide to Web Scraping without getting blocked in 2020
Crawlab
hakrawler
JobFunnel
You-Get
Universal Reddit Scraper
Gerapy
Ask HN: Best practices for ethical web scraping? (2020)
Newscatcher
Code
scrapio
Colly
Python Web Scraping with Virtual Private Networks (2020)