Getting Started with Impala: Interactive SQL for Apache Hadoop by John Russell

Author:John Russell [Russell, John]
Language: eng
Format: mobi, pdf
ISBN: 9781491905777
Publisher: O'Reilly Media
Published: 2014-09-24T16:00:00+00:00

Understanding Cluster Topology

As a developer, you might work with a different cluster setup than is actually used in production. Here are some things to watch out for, to understand the performance and scalability implications as your application moves from a dev/test setup into production:

For basic functional testing, you might use a single-node setup, perhaps running inside a virtual machine. You can check SQL compatibility, try out built-in functions, check data type compatibility and experiment with CAST(), see that your custom UDFs work correctly, and so on. (Perhaps with relatively small data volume, just to check correctness.)

To see what happens with distributed queries, you could use a relatively small cluster, such as two or four nodes. This allows you to see some performance and scalability benefits from parallelizing the queries. On a dev/test cluster, the name node is probably on the same host as one of the data nodes, which is not a problem when the cluster is running under a light workload.

For production, you’ll probably have a separate host for the name node, and a substantial number of data nodes. Here, the chances of a node failing are greater. (In this case, rerun any queries that were in flight.) Or one node might experience a performance issue, dragging down the response time of queries. (This type of problem is best detected with monitoring software such as Cloudera Manager.) Also, this is the time to double-check the guideline about installing Impala on all the data nodes in the cluster (to avoid I/O slowdown due to remote reads) and only on the data nodes (to avoid using up memory and CPU unnecessarily on the name node, which has a lot of work to do on a busy cluster).

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.