Inner Modules#

Sub-pacakge of util features

In design of distributed system or cluster, it always depends on some third party applications like message queue system (e.g., Kafka) or distributed system management, e.g., Zookeeper. So it has implementations is relative with that, but in generally, it won’t provide them to outside to use or extend. That’s the reason why this inner sub-package exists.

Here are some util functions like Zookeeper client or data processing of serializing or deserializing. Below functions or objects be encapsulated again to be more convenience and readable in usage.

Utils for inner usage in package#

smoothcrawler_cluster._utils.parse_timer(timer: str) → int | float[source]#

Parse the sleep time value to be an int or float type value.

Parameters:

timer (str) – The timer value to parse. Its time unit could be hour, minute or second.

Returns:

How many seconds it equals to. And it would return as float type value if it has decimal.

Return type:

Union[int, float]

Raises:

  • ValueError (not number format) – Invalid value which is NOT number format value so that it cannot be parsed.

  • ValueError (incorrect last character) – The last character is NOT correct so that it couldn’t judge its time unit is hour, minute or second.

class smoothcrawler_cluster._utils.MetaDataUtil(converter: BaseConverter, client: ZookeeperClient | None = None)[source]#

Encapsulate the util features

Integrate the util features, includes Zookeeper client and data processing of converting as another object which is more convenience and readable to use.

Parameters:

  • converter (BaseConverter) – The converter to do data processing.

  • client (Optional[ZookeeperClient]) – The Zookeeper client.

get_metadata_from_zookeeper(path: str, as_obj: Type[_BaseMetaDataType], must_has_data: bool = True) → Generic[_BaseMetaDataType][source]#

Get meta-data from Zookeeper.

Parameters:

  • path (str) – The node path.

  • as_obj (Type[_BaseMetaDataType]) – The target object it deserializes value to be.

  • must_has_data (bool) – If it’s True, it must return an object as the type it set by argument as_obj. In the other words, it would return an empty meta-data object if it gets None from Zookeeper. If it’s False, it would return None if it gets nothing from Zookeeper. Default value is True.

Returns:

The meta-data which has been deserialized as _BaseMetaData type instance from

Zookeeper.

Return type:

Generic[_BaseMetaDataType]

Raises:

TypeError – If the value type of argument as_obj DOES NOT one of these 4 types meta-data: GroupState, NodeState, Task and Heartbeat.

set_metadata_to_zookeeper(path: str, metadata: Generic[_BaseMetaDataType], create_node: bool = False) → None[source]#

Set value of meta-data to node by path in Zookeeper.

Parameters:

  • path (str) – The node path.

  • metadata (Generic[_BaseMetaDataType]) – The meta-data object to set to Zookeeper.

  • create_node (bool) – If it’s True, it would create the node with value. Nor it would set the value to node directly. Default value is False.

Returns:

None

For more details about util modules, here are some references: