Utils - Converter#

Util functions of serializing and deserializing of data from Zookeeper

No matter operating data as getting from or setting to Zookeeper, it must be byte or string type value. Therefore, the serializing or deserializing features would be deeply necessary in this package. Here module for all the features about serializing and deserializing.

JsonStrConverter#

class smoothcrawler_cluster._utils.converter.JsonStrConverter[source]#

Operating with JSON format data

It would operate with JSON format data to do serialize or deserialize, in the other words, it would try to serialize json object to be string type value, or deserialize JSON format string data as json object in Python.

_convert_to_str(data: Any) → str[source]#

Serialize the data as string type value from an Any type object.

Parameters:: data (Any) – Target data to serialize.
Returns:: The serializing result. It must be a string type value.
Return type:: str

_convert_from_str(data: str) → Any[source]#

Deserialize the data to Any type object from string type value.

Parameters:: data (str) – A string type value to deserialize.
Returns:: The final object which be deserialized. It could be customized object.
Return type:: Any

_convert_to_readable_object(obj: Generic[_BaseMetaDataType]) → Any[source]#

It would converse the object as dict type value (converse to JSON format data before serializing) via the common function to_readable_object which be defined in base class of meta-data.

Parameters:

obj (Generic[_BaseMetaDataType]) – The _BaseMetaData type object.

Returns:

Any type which could be serialized as string type value. In generally, it could be dict type which: mostly like JSON format value.

Return type:

Any

_convert_to_group_state(state: GroupState, data: Any) → GroupState[source]#

Converse the data as GroupState type instance.

Parameters:

state (GroupState) – The instance of GroupState.
data (Any) – The target data to deserialize.

Returns:

The meta-data GroupState instance keeps the values be deserialized from Zookeeper.

Return type:

GroupState

_convert_to_node_state(state: NodeState, data: Any) → NodeState[source]#

Converse the data as NodeState type instance.

Parameters:

state (NodeState) – The instance of NodeState.
data (Any) – The target data to deserialize.

Returns:

The meta-data NodeState instance keeps the values be deserialized from Zookeeper.

Return type:

NodeState

_convert_to_task(task: Task, data: Any) → Task[source]#

Converse the data as Task type instance.

Parameters:

task (Task) – The instance of Task.
data (Any) – The target data to deserialize.

Returns:

The meta-data Task instance keeps the values be deserialized from Zookeeper.

Return type:

Task

_convert_to_heartbeat(heartbeat: Heartbeat, data: Any) → Heartbeat[source]#

Converse the data as Heartbeat type instance.

Parameters:

heartbeat (Heartbeat) – The instance of Heartbeat.
data (Any) – The target data to deserialize.

Returns:

The meta-data Heartbeat instance keeps the values be deserialized from Zookeeper.

Return type:

Heartbeat

deserialize_meta_data(data: str, as_obj: Type[_BaseMetaDataType]) → Generic[_BaseMetaDataType]#

Deserialize the string type value to be one specific type object in Python.

Parameters:

data (str) – The string type value to deserialize.
as_obj (Type[_BaseMetaDataType]) – The target object it deserializes value to be.

Returns:

The meta-data instance from deserializing. It would be the instance of object: type from argument as_obj, i.e., it would return GroupState instance if value of argument as_obj is it.

>>> metadata = <BaseConverter type instance>.deserialize_meta_data(data=data, as_obj=GroupState)
>>> type(metadata)
<class 'GroupState'>

Return type:

Generic[_BaseMetaDataType]

Raises:

TypeError – If the value type of argument as_obj DOES NOT one of these 4 types meta-data: GroupState, NodeState, Task and Heartbeat.

serialize_meta_data(obj: Generic[_BaseMetaDataType]) → str#

Serialize data as string type value.

Parameters:: obj (Generic[_BaseMetaDataType]) – Target object to serialize.
Returns:: The serializing result. It must be a string type value.
Return type:: str

Other Modules#

TaskContentDataUtils#

class smoothcrawler_cluster._utils.converter.TaskContentDataUtils[source]#

Detail data in meta-data

For some meta-data, e.g., Task, it has a little bit complex meta-data structure. For example, its option running_result is also a dict type value, options running_content and result_detail even a list which element type is dict in list. For more clear and better to develop and maintain, it has namedtuple object to do data processing. And this util class for conversing dict type value as namedtuple object.

static convert_to_running_content(data: dict) → RunningContent[source]#

Converse data to namedtuple RunningContent.

Parameters:: data (dict) – The data which be keep by option running_content in Task meta-data.
Returns:: A namedtuple type object.
Return type:: RunningContent

static convert_to_running_result(data: dict) → RunningResult[source]#

Converse data to namedtuple RunningResult.

Parameters:: data (dict) – The data which be keep by option running_result in Task meta-data.
Returns:: A namedtuple type object.
Return type:: RunningResult

static convert_to_result_detail(data: dict) → ResultDetail[source]#

Converse data to namedtuple RunningContent.

Parameters:: data (dict) – The data which be keep by option result_detail in Task meta-data.
Returns:: A namedtuple type object.
Return type:: RunningContent