Machine Learning Projects for .NET Developers by Mathias Brandewinder

Author:Mathias Brandewinder
Language: eng
Format: epub, pdf
Publisher: Apress, Berkeley, CA

Getting to Know Our Data

Given the exploratory nature of our task, we will, once again, work from the F# scripting environment. Let’s create a new solution with an F# library project, Unsupervised, and, for convenience, add the data file to the solution itself. The dataset is in the form of a text file, userprofiles-toptags.txt, and can be downloaded from the following link: http://1drv.ms/1M727fP . If you take a look at it from Visual Studio, you should see something like this:

UserID,.net,ajax,android,arrays,asp.net,asp.net-mvc,c,c#,c++,css,django,html,ios,iphone,java,javascript,jquery,json,linux,mysql,objective-c,php,python,regex,ruby,ruby-on-rails,sql,sql-server,wpf,xml

1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

1000343,0,0,0,0,0,0,0,0,0,3,0,5,0,0,0,0,0,0,0,0,0,0,2,52,0,0,0,0,0,0

100297,0,0,0,26,0,0,0,0,0,0,99,62,0,0,0,29,0,182,0,26,0,0,4478,172,0,0,32,0,0,27

100342,0,0,0,0,0,1,0,0,0,0,0,3,0,0,0,16,5,0,0,0,0,0,0,0,1,7,0,0,0,0

The first row is a header describing the contents of each column. The first column contains the user ID, followed by 30 comma-separated columns, each of which contains the level of activity for that particular user and tag.

Before diving into algorithms, let’s start with basic statistics to get a sense for the lay of the land. We’ll begin with opening the Script.fsx file. We will read every line, drop the user IDs (which we don’t really need), and parse each line into an array of floats. We could also keep the values as integers, but as we are likely going to perform operations such as averages, we might as well directly convert to an easy-to-work-with type. We will also keep the headers in a separate array so that we can later map columns to the proper tag name.

Listing 5-1. Reading the dataset in memory

open System

open System.IO

let folder = __SOURCE_DIRECTORY__

let file = "userprofiles-toptags.txt"

let headers,observations =

let raw =

folder + "/" + file

|> File.ReadAllLines

// first row is headers, first col is user ID

let headers = (raw.[0].Split ',').[1..]

let observations =

raw.[1..]

|> Array.map (fun line -> (line.Split ',').[1..])

|> Array.map (Array.map float)

headers,observations

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.