Advanced Analytics with Spark: Patterns for Learning from Data at Scale by Sandy Ryza & Uri Laserson & Sean Owen & Josh Wills

Author:Sandy Ryza & Uri Laserson & Sean Owen & Josh Wills [Ryza, Sandy]
Language: eng
Format: azw3
Publisher: O'Reilly Media
Published: 2017-06-12T04:00:00+00:00

Finding Important Concepts

So SVD outputs a bunch of numbers. How can we inspect these to verify they actually relate to anything useful? The V matrix represents concepts through the terms that are important to them. As discussed earlier, V contains a column for every concept and a row for every term. The value at each position can be interpreted as the relevance of that term to that concept. This means that the most relevant terms to each of the top concepts can be found with something like this:

import org.apache.spark.mllib.linalg.{Matrix, SingularValueDecomposition} import org.apache.spark.mllib.linalg.distributed.RowMatrix def topTermsInTopConcepts( svd: SingularValueDecomposition[RowMatrix, Matrix], numConcepts: Int, numTerms: Int, termIds: Array[String]) : Seq[Seq[(String, Double)]] = { val v = svd.V val topTerms = new ArrayBuffer[Seq[(String, Double)]]() val arr = v.toArray for (i <- 0 until numConcepts) { val offs = i * v.numRows val termWeights = arr.slice(offs, offs + v.numRows).zipWithIndex val sorted = termWeights.sortBy(-_._1) topTerms += sorted.take(numTerms).map { case (score, id) => (termIds(id), score) } } topTerms }

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.