Hands-On Software Engineering with Golang by Achilleas Anagnostopoulos

Author:Achilleas Anagnostopoulos [Achilleas Anagnostopoulos]
Language: eng
Format: epub
Tags: COM051230 - COMPUTERS / Software Development and Engineering / General, COM051010 - COMPUTERS / Programming Languages / General, COM051240 - COMPUTERS / Software Development and Engineering / Systems Analysis and Design
Publisher: Packt
Published: 2020-01-24T05:32:02+00:00

The Config type holds all required configuration options for creating a new crawler pipeline:

// Config encapsulates the configuration options for creating a new Crawler. type Config struct { PrivateNetworkDetector PrivateNetworkDetector URLGetter URLGetter Graph Graph Indexer Indexer FetchWorkers int }

The caller of the crawler's constructor is expected to provide the following configuration options:

An object that implements the PrivateNetworkDetector interface, which will be used by the link fetcher and link extractor components to filter out links that resolve to private network addresses

An object that implements the URLGetter interface (for example, http.DefaultClient), which the link fetcher will use to perform HTTP GET requests

An object that implements the Graph interface (for example, any of the link graph implementations from the previous chapter), which the graph updater component will use to upsert discovered links into the link graph

An object that implements the Indexer interface (for example, any of the indexer implementations from the previous chapter), which the text indexer component will use to keep the search index in sync

The size of the worker pool for executing the link fetcher stage of the pipeline

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.