Source code for smoothcrawler_cluster.model.metadata_enum

"""*Enum of meta-data which has multiple fixed values*

For some meta-data options, e.g., *role* of **NodeState**, it has multiple fixed values as signal at it, and it won't
let outside modify it or assign it to meta-data casually. The module provides some enum objects to usage of meta-data
to let value could be more clear and expected.
"""

from enum import Enum


[docs]class CrawlerRole(Enum): """*The crawler role in crawler cluster system* This role is NOT the role of *SmoothCrawler-AppIntegration*. They're very different. The role in *SmoothCrawler-AppIntegration* means source site (or producer) of application or processor site (or consumer) of application. But the role in meta-data in *SmoothCrawler-Cluster* means it is active runner to run task or backup of that active runner. For **SmoothCrawler-Cluster** realm, it has 4 different roles: * Initial * Runner * Backup Runner * Dead Runner * Dead Backup Runner """ INITIAL = "initial" """It only have this state in instantiating process of crawler and before runner election. And it would change this option to *Runner* or *Backup_Runner* after done runner election. """ RUNNER = "runner" """Literally, **Runner** role is the major element to run web spider tasks.""" BACKUP_RUNNER = "backup-runner" """**Backup Runner** role is the backup of **Runner**. It would activate (base on the setting, it may activate immediately) and run the web spider task if it still not finish. A **Backup Runner** would keep checking the heartbeat info of **Runner**, standby and ready to run in anytime for any one of **Runner** does not keep updating its own heartbeat info (and it would turn to **Dead Runner** at that time). """ DEAD_RUNNER = "dead-runner" """If **Runner** cannot work finely, like the entire VM be shutdown where the crawler runtime environment in. It would turn to be **Dead Runner** from **Runner**. In other words, it must to be **Dead Runner** if it cannot keep updating its own heartbeat info. """ DEAD_BACKUP_RUNNER = "dead-backup-runner" """**Dead Backup Runner** is same as **Dead Runner** but it's for **Backup Runner**."""
[docs]class TaskState(Enum): """*The task running state* The task result means it is the result of running web spider task. The web spider task could classify to be 4 different states: processing, done, terminate and error. """ NOTHING = "nothing" """When crawler just be ready for running or in instantiating process, It would be this state.""" PROCESSING = "processing" """Task running in processing.""" DONE = "done" """Finish the task and it works finely without any exceptions.""" TERMINATE = "terminate" """Web spider task running has been terminated so that it cannot finish all processes, but it doesn't occur any exceptions except KeyboardInterrupt. """ ERROR = "error" """If it raise any exceptions in web spider task running, its result would be error."""
[docs]class HeartState(Enum): """*State of heartbeat at that time* The heartbeat state. """ NEWBORN = "Newborn" """The initialized heartbeat state of a crawler instance which just be created, in other words, crawler instance's heartbeat state would be this before it run updating heartbeat process. """ HEALTHY = "Healthy" """This means that current crawler instance runs finely without any issue.""" ARRHYTHMIA = "Arrhythmia" """It means that the current crawler instance is late to update property *heart_rhythm_time*. It likes its heartbeat rate is not stable and cannot work normally lead to it looks or sounds not good, just like arrhythmia in human. """ ASYSTOLE = "Asystole" """If a crawler instance's property *healthy_state* become this state, it means the crawler be judged it's dead by other alive crawler instances. So the dead crawler instance won't run anymore and list to *fail_xxx* property in *State*. """ APPARENT_DEATH = "Apparent Death" """It would be this state if the crawler instance would be stopped or killed manually."""