Group State#

class smoothcrawler_cluster.model.metadata.GroupState[source]#

Meta-Data for group in cluster

One of the meta-data of SmoothCrawler-Cluster. It saves info about which VMs (web spider name) are Runner and another VMs are Backup Runner. The cluster would check the content of this info to run Runner Election to determine who is Runner and who is *Backup Runner.

Note

It could consider one thing: use 2 modes to determine how cluster works:

  1. Decentralized mode: no backup member, but each member could cover anyone which is dead.

  2. Master-Slave mode: has backup member, and they do nothing (only check heartbeat) until anyone dead.

  • Zookeeper node path:

/smoothcrawler/group/<crawler group name>/state/

  • Example data:

{
    "total_crawler": 3,
    "total_runner": 2,
    "total_backup": 1,
    "current_crawler": ["spider_1", "spider_2", "spider_3"],
    "current_runner": ["spider_1", "spider_2"],
    "current_backup": ["spider_3"],
    "standby_id": "3",
    "fail_crawler": [],
    "fail_runner": [],
    "fail_backup": []
}
to_readable_object() → dict[source]#

Converse the instance’s current data to be dict type value. Its target is let data converse as JSON format value for deserializing conveniently.

Returns:

A dict type value keeps the current instance’s data.

Return type:

dict

property total_crawler: int#

Properties with both a getter and setter for total amount of the crawler includes role is Runner & Backup Runner.

Setter would block the data which type is NOT int.

Type:

int

property total_runner: int#

Properties with both a getter and setter for total amount of crawler with role Runner.

Setter would block the data which type is NOT int.

Type:

int

property total_backup: int#

Properties with both a getter and setter for total amount of crawler with role Backup Runner.

Setter would block the data which type is NOT int.

Type:

int

property current_crawler: List[str]#

Properties with both a getter and setter for CURRENTLY total of Runner & Backup Runner as a list of their name.

Setter would block the data which type is NOT list.

Type:

list of str

property current_runner: List[str]#

Properties with both a getter and setter for CURRENTLY total amount of Runner as a list of their name.

Setter would block the data which type is NOT list.

Type:

list of str

property current_backup: List[str]#

Properties with both a getter and setter for CURRENTLY total amount of Backup Runner as a list of their name.

Setter would block the data which type is NOT list.

Type:

list of str

property standby_id: str#

Properties with both a getter and setter for the next one member of Backup Runner should activate to take over the task of Runner member which is dead (doesn’t update heartbeat stamp and timeout).

Setter would block the data which type is NOT str.

Type:

str

property fail_crawler: List[str]#

Properties with both a getter and setter for CURRENTLY total amount of Dead Runner & Dead Backup Runner as a list of their name.

Setter would block the data which type is NOT list.

Type:

list of str

property fail_runner: List[str]#

Properties with both a getter and setter for CURRENTLY total amount of Dead Runner as a list of their name.

Setter would block the data which type is NOT list.

Type:

list of str

property fail_backup: List[str]#

Properties with both a getter and setter for CURRENTLY total amount of Dead Backup Runner as a list of their name.

Setter would block the data which type is NOT list.

Type:

list of str