Abstract Crawlers#
Base Distributed Crawler#
- class smoothcrawler_cluster.crawler.crawlers.BaseDistributedCrawler[source]#
Base class for distributed crawler
The base class about distributed crawler definitions. Distributed system has many practices, i.e., using application integration concern to design to handle complex business logic in a large system; using cluster concern to build a crawler system which has high fault tolerance to support each other’s feature, etc. Therefore, this is the most basically class for defining a distributed crawler.
Base Cluster Crawler#
- class smoothcrawler_cluster.crawler.crawlers.BaseClusterCrawler[source]#
Base class for cluster crawler
The base class about cluster crawler definitions. This is the most basically class for the cluster crawler which has high fault tolerance feature.
In cluster realm, it could roughly divide to 2 types: Centralized and Decentralized. The further one has leader role member(s) and the letter one doesn’t. So it has one more sub-abstract classes of this one are BaseCentralizedCrawler and BaseDecentralizedCrawler.
In generally, each of single instance in cluster all are a single one which could be standby for each other and hand over other’s jobs if it needs. The consideration is: how could it work? what roles or jobs it need? We will have discussion of it with the 2 abstract classes: BaseCentralizedCrawler and BaseDecentralizedCrawler.
Base Centralized Crawler#
- class smoothcrawler_cluster.crawler.crawlers.BaseCentralizedCrawler[source]#
Base class for centralized cluster crawler
The base class about centralized crawler definitions. For centralized system, it has leader (or be called as master or something else like that) role in it.
Please refer to Has Leader to get more details.
Base Decentralized Crawler#
- class smoothcrawler_cluster.crawler.crawlers.BaseDecentralizedCrawler[source]#
Base class for decentralized cluster crawler
The base class about decentralized crawler definitions. For decentralized system, it doesn’t have leader, master or something else like that role in it.
Please refer to No Leader to get more details.