Apache Spark Quick Start Guide by Shrey Mehrotra

Author:Shrey Mehrotra
Language: eng
Format: epub, azw3
Tags: COM018000 - COMPUTERS / Data Processing, COM062000 - COMPUTERS / Data Modeling and Design, COM021030 - COMPUTERS / Databases / Data Mining
Publisher: Packt Publishing
Published: 2019-01-31T19:55:06+00:00

sales_df.write.json("sales.json")

val json_sales_DF = spark.read.json("sales.json")

json_sales_DF.createOrReplaceTempView("jsonSales")

var ipDF = spark.sql("SELECT ip FROM jsonSales WHERE id BETWEEN 10 AND 19")

ipDF.map(attributes => "IPS: " + attributes(0)).show()

DataFrame operations and associated functions

DataFrames support untyped transformations with the following operations:

printSchema: This prints out the mapping for a Spark DataFrame in a tree structure. The following code will give you a clear idea of how this operation works:

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.