Hadoop Real-World Solutions Cookbook - Second Edition by Tanmay Deshpande

Author:Tanmay Deshpande [Deshpande, Tanmay]
Language: eng
Format: azw3, mobi, epub
Publisher: Packt Publishing
Published: 2016-03-31T04:00:00+00:00

Total MapReduce CPU Time Spent: 1 seconds 700 msec OK 1 Tanmay 3500.0 101 101 Engineering 2 Sneha 5599.0 101 101 Engineering 3 Avinash 6700.0 102 102 HR 4 Manisha 6400.0 103 103 Finance 5 Sakalya 3200.0 102 102 HR Time taken: 17.748 seconds, Fetched: 5 row(s)

The preceding execution shows that Hive informs you when it has converted this join into a map side join.

How it works...

Map side joins are performed when we have one small table and a very big table join. While executing, Hive reads the small table completely into memory and writes this table to a hash table. This hash table is referenced by each mapper to join the bigger table. By getting one table in its memory, Hive has fewer disk I/O operations, and this means less time to get records. There are limits that are set in the Hive configurations that decide whether the table is small or not. The hive.auto.convert.join.noconditionaltask and hive.auto.convert.join.noconditionaltask.size configurations decide the auto conversion of joins and the size of the table that can fit into the memory.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.