src.producer module

Producer is responsible for generating arbitrary streams of data and placing it onto a provided queue. Producer is designed to execute indefinitely until terminated by a parent Crawler.

class src.producer.Producer(id: int, source: Queue, sink: Queue, transformer: Transformer, event_bus: EventBus)

Bases: object

async run()

Run the Producer — i.e, read from the self.source; process items; and feed back to self.sink until terminated by parent Crawler.

async spread(results: List[Any]) → None

Place each link in the list of results onto the input queue for currently running Consumers.

Parameters:

results (List[str]) – A list of elements collected from the previously processed element.

Returns:

Void.

Return type:

None