Scaling Machine Learning with Spark by Adi Polak

Author:Adi Polak [Adi Polak]
Language: eng
Format: epub
Publisher: O'Reilly Media, Inc.
Published: 2023-02-25T00:00:00+00:00

There are many other filter features Pillow provides us with out-of-the-box such as BLUR, SMOOTH, EDGE_ENHANCE, and others. all of which are based on adding a filter to the image based on pixel gradient manipulation. Figure 3-7 captures how grey-scale and edge-filter features are rendered.

Extracting Features Leveraging Spark APIs

Let’s examine multiple techniques using the Caltech256 dataset. In the preprocessing chapter, we started touching a bit on Python UDFs and how to use them for extracting image size. Let’s dive deeper into this topic as it is our main tool for feature extraction over images.

Until 2017, PySpark supported Python UDFs that operated on one-row-at-at-a-time, those functions missed on Spark Query Engine optimization capabilities and since, behind the scenes, PySpark is being translated into Scala code, many UDFs written in PySpark had high serialization and invocation overhead. As a result, data engineers and data scientists worked together to define UDFs in Java and Scala where data scientists invoke them from Python. This made the code messy, hard to navigate and maintain. Fortunately, Spark 3.0 introduced Pandas UDFs together with Python Type Hints that allow us to run on groups of rows at a time, leverage Apache Arrow optimization for reducing deserialization operations, and use Pandas API from within the function.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.