Flow

For understanding how a crawler cluster run well, it’s a great idea of explaining it with some clear work flow. This section would explain the details of how one or more group(s) of crawlers could run finely with each others as a cluster.

It divides this part to 2 sections because the cluster working could be demonstrated in 2 different ways, one is the general way — how it work finely, so explain it with work flow; but it also could be explain with the data changing in cluster working — how it work finely with the meta-data, so also could explain it with data flow.

Please refer to below 2 sections: Work Flow and Data Flow to get more details:

• Work Flow
  • Life Cycle
• Data Flow
  • Data pipline with role