Big Data Processing with Apache Spark by Manuel Ignacio Franco Galeano
Author:Manuel Ignacio Franco Galeano
Language: eng
Format: epub
Published: 2018-10-30T00:00:00+00:00
Write a command to start the TCP server in port 12345:python3 log_socket_producer.py --port 12345
Open another two windows and start another two instances of this server in ports 9876 and 8765.
Creating a TCP Spark Stream Consumer
Create a Python file named log_socket_producer.py and import the necessary packages and global variables:from pyspark import SparkContext
from pyspark.streaming import StreamingContext
import argparse
import re
import os
Write a function that applies a regular expression to every incoming message and returns a dictionary:def parse_log_entry(msg):
"""
Parse a log entry from the format
$ip_addr - [$time_local] "$request" $status $bytes_sent $http_user_agent"
to a dictionary
"""
data = {}
# Regular expression that parses a log entry
search_term = '(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s+\-\s+\[(.*)]\s+'\
'"(\/[/.a-zA-Z0-9-]+)"\s+(\d{3})\s+(\d+)\s+"(.*)"'
values = re.findall(search_term, msg)
if values:
val = values[0]
data['ip'] = val[0]
data['date'] = val[1]
data['path'] = val[2]
data['status'] = val[3]
data['bytes_sent'] = val[4]
data['agent'] = val[5]
return data
Download
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.
Access | Data Mining |
Data Modeling & Design | Data Processing |
Data Warehousing | MySQL |
Oracle | Other Databases |
Relational Databases | SQL |
Algorithms of the Intelligent Web by Haralambos Marmanis;Dmitry Babenko(8293)
Azure Data and AI Architect Handbook by Olivier Mertens & Breght Van Baelen(6675)
Building Statistical Models in Python by Huy Hoang Nguyen & Paul N Adams & Stuart J Miller(6651)
Serverless Machine Learning with Amazon Redshift ML by Debu Panda & Phil Bates & Bhanu Pittampally & Sumeet Joshi(6524)
Data Wrangling on AWS by Navnit Shukla | Sankar M | Sam Palani(6312)
Driving Data Quality with Data Contracts by Andrew Jones(6263)
Machine Learning Model Serving Patterns and Best Practices by Md Johirul Islam(6029)
Learning SQL by Alan Beaulieu(5987)
Weapons of Math Destruction by Cathy O'Neil(5775)
Big Data Analysis with Python by Ivan Marin(5331)
Data Engineering with dbt by Roberto Zagni(4332)
Solidity Programming Essentials by Ritesh Modi(3977)
Time Series Analysis with Python Cookbook by Tarek A. Atwan(3834)
Pandas Cookbook by Theodore Petrou(3545)
Blockchain Basics by Daniel Drescher(3292)
Hands-On Machine Learning for Algorithmic Trading by Stefan Jansen(2900)
Feature Store for Machine Learning by Jayanth Kumar M J(2808)
Learn T-SQL Querying by Pam Lahoud & Pedro Lopes(2791)
Mastering Python for Finance by Unknown(2742)
