Inner Data

Data for inner usage

Here provides some data objects for inner usage in SmoothCrawler-Cluster.

New in version 0.2.0.

class smoothcrawler_cluster.model._data.CrawlerName

Data about crawler’s name

property group: str

Properties with both getter and setter for which cluster group the crawler instance is in.

Type:

str

property base_name: str

Properties with both getter and setter for crawler instance’s name part. Its value could be repeated, but the combination of base_name and id should be unique to let cluster could identify it. For example, a crawler instance’s name would be like sc-crawler_1, the sc-crawler is base_name part, and the _ is the index separation, 1 is its ID.

Type:

str

property id: str

Properties with both getter and setter for the identity of crawler instance. It MUST be unique so that it could combine with base_name and index_separation as an entire unique crawler’s name which could be identified by cluster.

Type:

str

property index_separation: str

Properties with both getter and setter for parsing to get index info from current crawler instance’s name.

Type:

str

class smoothcrawler_cluster.model._data.CrawlerTimer

Data about crawler’s time attributes, e.g., time interval, threshold, etc.

property time_interval: TimeInterval

Properties with both getter and setter. This property is TimeInterval object. Please refer to TimeInterval to get more info.

Type:

TimeInterval

property threshold: TimerThreshold

Properties with both getter and setter. This property is TimerThreshold object. Please refer to TimerThreshold to get more info.

Type:

TimerThreshold

class smoothcrawler_cluster.model._data.TimeInterval

Data about each different time interval

property check_task: float

Properties with both getter and setter. It is the interval of checking whether the current crawler receives any task or not. This property for Runner.

Type:

float

property check_crawler_state: float

Properties with both getter and setter. It is the interval of checking whether it has anyone of all current crawlers is dead or not. This property for primary Backup Runner.

Type:

float

property check_standby_id: float

Properties with both getter and setter. It is the interval of checking whether the current crawler could be the primary backup one or not. This property for secondary BackupRunner.

Type:

float

class smoothcrawler_cluster.model._data.TimerThreshold

Data about threshold of time

property reset_timeout: int

Properties with both getter and setter. It is the threshold of reset timeout record for checking all the current crawler’s heartbeat state. This property for primary Backup Runner.

Type:

int