apache flume tutorial - Wikitechy

How Flume used with Hbase ?

Editor — Sun, 11 Jul 2021 16:20:07 +0000

Apache Flume used with Hbase

- Apache Flume can be used with HBase utilizing one of the two HBase sinks –
  - HBaseSink (org.apache.flume.sink.hbase.HBaseSink) – It supports protected HBase clusters and furthrtmore the novel HBase IPC that was presented in the version HBase 0.96.
  - AsyncHBaseSink (org.apache.flume.sink.hbase.AsyncHBaseSink) – It has improved performance than HBase sink as it can simply make non-blocking calls to HBase.

Working of the HBaseSink

- - In HBaseSink, a Flume Event is changed over into HBase Increments or Puts. Serializer executes the HBaseEventSerializer which is then instantiated when the sink begins.
  - For each and every event, sink calls the initialize method in the serializer which then translates the Flume Event into HBase increments and puts to be sent to HBase cluster.

Working of the AsyncHBaseSink

- This sink executes the AsyncHBaseEventSerializer. The initialize method is called just once by the sink when it begins.
- The sink invokes the setEvent method and then makes calls to the getIncrements and getActions methods just similar to HBase sink. When the sink pause, the cleanUp method is called by the serializer.

What are the channel types in Flume and Which channel type is faster ?

Editor — Sun, 11 Jul 2021 16:10:38 +0000

Channels in Flume

It stores events; events are delivered to the channel by means of sources operating within the agent.
An event remains in the channel until a sink removes it for further transport.

There are three different built in channels in Flume –

MEMORY Channel – Events are study from the source into memory and passed to the sink.
JDBC Channel – It stores the events in an embedded Derby database.
FILE Channel – It writes the contents to a file on the file system after reading the event from a source. The file is erased only after the contents are successfully delivered to the sink.

MEMORY Channel – It is the fastest channel among the three however has the risk of data loss. The channel that you choose totally depends on the nature of the big data application and the value of each event.

What is Interceptor in Apache Flume ?

Editor — Sun, 11 Jul 2021 15:29:39 +0000

Interceptor in Apache Flume

Apache Flume offers interceptors as a method of modifying records as they have a Flume channel.
Interceptors are used to filter the events between source and channel, channel and sink. These channels will filter un-necessary or targeted log files.

Interceptors are a part of Flume’s extensibility model. They permit events to be inspected as they pass between a source and a channel, and also the developer is liberated to modify or drop events as needed. Interceptors in the chain along to create a pipeline process.

What is Apache flume ?

Editor — Sun, 11 Jul 2021 15:17:38 +0000

Apache flume

Apache Flume is a reliable, distributed and accessible service for efficiently aggregating, collecting, and moving huge amounts of log data.
It has an easy and flexible design based on streaming data flows. It’s robust and fault tolerant with tunable reliability mechanisms and recovery mechanisms.
It uses an easy extensible data model that enables for on-line analytic application.

Learn Flume – Flume tutorial – apache flume – Flume examples – Flume programs

Flume defines a simple pipeline structure with three roles:

1. Source
2. Channel
3. Sinks

Sources define where data comes from, e.g. a file, a message queue (Kafka,JMS).
Channels are pipes connecting sources with sinks.
Sinks are the destination of the data pipelined from sources.