Pig Design Patterns by 未知

Author:未知
Language: eng
Format: epub
Publisher: Packt Publishing

Motivation

The basic statistical profiling design pattern helps to create data quality metadata that includes basic statistics, such as mean, median, mode, maximum, minimum, and standard deviation. These stats give you a complete snapshot of the entire data field, and tracking these statistics over time will give insights into the characteristics of new data that is being ingested by the Hadoop cluster. Basic statistics of new data could be checked before ingesting it into Hadoop, to be forewarned about the inconsistent data and help prevent adding low-quality data.

This design pattern tries to address the following profiling requirements:

Range analysis methods scan values and determine if the data is subject to a total ordering, and also determine whether the values are constrained within a well-defined range

The sparseness of the data can be evaluated to find the percentage of the elements that are not populated

The cardinality of the dataset can be analyzed by finding the number of distinct values that appear within the data

The uniqueness can be evaluated to figure out if each of the value assigned to the attribute is indeed exclusive

Overloading of the data can be evaluated to check if the attribute is being used for multiple purposes

Format evaluation can be done by resolving unrecognized data into defined formats

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.