Using Flume by Shreedharan Hari

Author:Shreedharan, Hari
Language: eng
Format: epub
Publisher: O'Reilly Media
Published: 2014-09-16T00:00:00+00:00

hdfs.roundValue

1

The timestamp will be rounded to the largest multiple of this parameter in the units specified by the hdfs.roundUnit parameter.

hdfs.roundUnit

second

The unit of the hdfs.roundValue configuration parameter (can be second, minute, or hour).

If the hdfs.round parameter is true, the HDFS Sink reads the timestamp from the event headers and rounds it down to the highest multiple of the value specified by hdfs.roundValue in the unit specified by hdfs.roundUnit. Rounded-down values are not used when the %t escape sequence is used. Let’s consider a sink with the configuration parameters shown here:

agent.sinks.hdfsSink.hdfs.path = /Data/Flume/%{topic}/%Y/%m/%d/%H/%M agent.sinks.hdfsSink.hdfs.round = true agent.sinks.hdfsSink.hdfs.roundUnit = minute agent.sinks.hdfsSink.hdfs.roundValue = 10

If an event with the timestamp equivalent of 2:35 PM on September 1, 2014 comes in, the HDFS Sink rounds the timestamp down to 2:30 PM on September 1, 2014 and writes the event out to the /Data/Flume/UsingFlume/2014/09/01/14/30 directory. In fact, every event between 2:30:00:00:000 PM and 2:39:59:59:999 will get written out to the same directory. Using a similar configuration mechanism (as shown next), the user can have buckets of two hours each:

agent.sinks.hdfsSink.hdfs.path = /Data/Flume/%{topic}/%Y/%m/%d/%H/%M agent.sinks.hdfsSink.hdfs.round = true agent.sinks.hdfsSink.hdfs.roundUnit = hour agent.sinks.hdfsSink.hdfs.roundValue = 2

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.