Contents Menu Expand Light mode Dark mode Auto light/dark mode
SmoothCrawler-Cluster 0.2.0 documentation
SmoothCrawler-Cluster 0.2.0 documentation

Contents:

  • Introduce
  • Installation
  • Quickly Start
  • Advanced Usage
    • Self-Define Utils
      • How to implement new Zookeeper client
      • How to write new Converter
    • How to customize new Election

Usage Guides:

  • No Leader
    • All are Runner
    • Some are Runner
  • Has Leader
    • Leader (Master)
    • Follower (Slave)
  • Handle Meta-Data
    • Depend on third party application - Zookeeper
    • Handle by itself

API Reference:

  • Crawler Sub-Package
    • Crawler Module
      • Abstract Crawlers
      • Zookeeper Crawlers
    • Attributes
    • Adapter
    • Dispatcher
    • Workflow
  • Register
    • General Registration
  • Election
    • Abstract Election
    • Smaller Election
  • Model
    • Meta-Data Modules
      • Abstract Meta-Data
      • Group State
      • Node State
      • Task
      • Heartbeat
    • Meta-Data Enum Objects
      • CrawlerRole
      • TaskState
      • HeartState
    • Inner Data
  • Inner Modules
    • Utils - Zookeeper
    • Utils - Converter

Development Documentation

  • Flow
    • Work Flow
    • Data Flow
  • Software Architecture
    • Crawler Sub-Package
      • Crawler
      • Workflow
      • Dispatcher
      • Attributes
    • Election
    • Utils
  • CI / CD

Change Logs

  • SmoothCrawler-Cluster 0.X
    • Version 0.X
Back to top
Edit this page

Crawler Module#

The crawler module at smoothcrawler_cluster.crawler.crawlers.

  • Abstract Crawlers
    • Base Distributed Crawler
    • Base Cluster Crawler
    • Base Centralized Crawler
    • Base Decentralized Crawler
  • Zookeeper Crawlers
    • ZookeeperCrawler
Next
Abstract Crawlers
Previous
Crawler Sub-Package
Copyright © 2022, Liu, Bryant
Made with Furo