Structured Lists Directory

YAML

Web Scraping Tools - YAML

Web scraping tools are software and libraries for automatically collecting structured data from websites on the internet. Various tools exist to accommodate different approaches and use cases, from static HTML parsing to dynamic JavaScript rendering and browser automation. Implemented in multiple programming languages including Python, JavaScript, and Java, they are widely used for everything from small-scale data collection to large-scale crawling projects.

web scraping data collection crawling automation Python JavaScript 
- code: "01"
  slug: "scrapy"
  name: "Scrapy"
  description: "A high-level Python web crawling and scraping framework"
  language: "Python"
  type: "Framework"
  javascriptSupport: false
  officialUrl: "https://scrapy.org/"
  githubUrl: "https://github.com/scrapy/scrapy"
- code: "02"
  slug: "beautifulsoup"
  name: "BeautifulSoup"
  description: "A Python library for parsing HTML and XML documents"
  language: "Python"
  type: "Library"
  javascriptSupport: false
  officialUrl: "https://www.crummy.com/software/BeautifulSoup/"
  githubUrl: null
- code: "03"
  slug: "selenium"
  name: "Selenium"
  description: "A cross-platform tool for browser automation"
  language: "Multi-language"
  type: "Framework"
  javascriptSupport: true
  officialUrl: "https://www.selenium.dev/"
  githubUrl: "https://github.com/SeleniumHQ/selenium"
- code: "04"
  slug: "playwright"
  name: "Playwright"
  description: "Microsoft's end-to-end testing and automation framework"
  language: "Multi-language"
  type: "Framework"
  javascriptSupport: true
  officialUrl: "https://playwright.dev/"
  githubUrl: "https://github.com/microsoft/playwright"
- code: "05"
  slug: "puppeteer"
  name: "Puppeteer"
  description: "Google's Node.js library for Chrome and Firefox automation"
  language: "JavaScript/Node.js"
  type: "Library"
  javascriptSupport: true
  officialUrl: "https://pptr.dev/"
  githubUrl: "https://github.com/puppeteer/puppeteer"
- code: "06"
  slug: "octoparse"
  name: "Octoparse"
  description: "A no-code visual web scraping tool"
  language: "N/A"
  type: "No-code Tool"
  javascriptSupport: true
  officialUrl: "https://www.octoparse.com/"
  githubUrl: null
- code: "07"
  slug: "apify"
  name: "Apify"
  description: "A cloud-based web scraping and automation platform"
  language: "JavaScript/Node.js"
  type: "Cloud Platform"
  javascriptSupport: true
  officialUrl: "https://apify.com/"
  githubUrl: "https://github.com/apify"
- code: "08"
  slug: "parsehub"
  name: "ParseHub"
  description: "A machine learning-powered cloud-based scraping tool"
  language: "N/A"
  type: "Cloud Tool"
  javascriptSupport: true
  officialUrl: "https://www.parsehub.com/"
  githubUrl: null