src.consumer module

The Consumer class is responsible for accepting and processing arbitrary streams of data from a provided queue. Consumer is designed to execute indefinitely until terminated by a parent Crawler.

class src.consumer.Consumer(id: int, source: Queue, sink: Queue, transformer: Transformer, event_bus: EventBus)

Bases: object

async run()

Run the Consumer — i.e, read from the self.source; process items; and feed back to self.sink until terminated by parent Crawler.

async spread(results: List[Any])

Place each element in the list of results onto the input queue for currently running Producers.

Parameters:

results (List[str]) – A list of elements collected from the previously processed element.

Returns:

Void.

Return type:

None