Learning Real-time Processing with Spark Streaming by Gupta Sumit

Author:Gupta, Sumit
Language: eng
Format: epub
Publisher: Packt Publishing
Published: 2015-09-27T16:00:00+00:00

We are done with our setup. Now the logs are being generated in real time and we need to consume them and perform further analysis.

Let us move to the next section where we will discuss the transformation operations provided by Spark Streaming.

Functional operations

Discretized streams or DStream are nothing but a series of RDD which are represented by org.apache.spark.streaming.dstream.DStream.scala and org.apache.spark.streaming.dstream.PairDStreamFunctions.scala. It defines various higher-order functions like map and reduce that accepts and apply a given function to each element of the RDD and produces a new RDD. As per Wikipedia, higher-order functions are those functions that either accept a function as input or output a function. It is a common concept, defined in functional programming languages like Clojure, Lisp, Erlang, Haskell and now in Java-8 too.

The following are the different higher-order functions and operations provided by DStream:

flatMap(flatMapFunc): DStream[U]—Similar to map(…) but, before returning, it flattens the results and then returns the final result set.

forEachRDD(forEachFunc): Applies the given function to all RDDs in a given stream. It is a special type of function and it is worth noting that the given function is applied to all RDDs on the driver node itself but there could be actions defined in the RDD and all those actions are performed over the cluster. forEach is categorized as an output operator and, by default, output operations are executed one-at-a-time, sequentially in the order they are defined in the application.

filter(filterFunc): DStream[T]—Applies the provided function to all the elements of RDD and generates the RDD only for those elements which return TRUE.

map(mapFunc): DStream[U]—Applies a given function mapFunc to all elements of RDD and generates a new RDD.

mapPartitions(mapPartFunc, preservePartitioning): Return a new DStream in which each RDD is generated by applying mapPartitions() to each RDD in the invoking DStream. Applying mapPartitions() to an RDD applies the given function to each partition of the RDD.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.