PySpark SQL Recipes by Raju Kumar Mishra & Sundar Rajan Raman

Author:Raju Kumar Mishra & Sundar Rajan Raman
Language: eng
Format: epub
ISBN: 9781484243350
Publisher: Apress

The results depict that the DataFrame has been repartitioned into two parts. Now it’s time to perform partition-wise sorting. But before we write the code for partition-wise sorting, let’s look at the process of sorting on repartitioned data.

Figure 4-9 displays the partition-wise sorting. Just concentrate on the left side and the right side Partition 1 DataFrame partition. In the process of sorting, nothing was shuffled. While sorting, each partition was considered an independent, full DataFrame.

Figure 4-9Partitioning and sorting

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.