Adapter#
The adapter of features which would be used in workflow
This module has adapters of some features like distributed lock, major processing function, etc. Workflow is responsible for defining processing detail how it works and what details it does. But for some features, e.g., distributed lock, it may be used and may not be used. In the other words, if the workflow processing with Zookeeper, it absolutely has distributed lock; if the workflow processing with each by algorithm, i.g., sync up the meta-data with each other by gossip algorithm, it doesn’t have and also doesn’t need to use distributed lock in the workflow. Therefore, the responsibility what thing it should do (whether it runs with lock or not, no matter it doesn’t have, or it doesn’t need) and how it works would let adapter module to handle.
New in version 0.2.0.
Distributed Lock#
- class smoothcrawler_cluster.crawler.adapter.DistributedLock(lock: Callable, *args, **kwargs)[source]#
Adapter of distributed lock feature
This is the adapter of distributed lock. It’s responsible for running target function with lock if it needs.
- Parameters:
- weakly_run(function: Callable, *args, **kwargs) Any[source]#
Try to run the target function synchronously with lock. If the lock function doesn’t have special methods one of __enter__ and __exit__, it would keep running the function directly without lock.
- strongly_run(function: Callable, *args, **kwargs) Any[source]#
Try to run the target function synchronously with lock. The lock function must have both of special methods __enter__ and __exit__, nor it would raise an exception to it.
- Parameters:
- Returns:
The return value of the target function.
- Raises:
NotImplementedError – If the lock function doesn’t have special methods __enter__ and __exit__. It would raise this exception.