• • Apache Flume can be used with HBase utilizing one of the two HBase sinks –
    • HBaseSink (org.apache.flume.sink.hbase.HBaseSink) – It supports protected HBase clusters and furthrtmore the novel HBase IPC that was presented in the version HBase 0.96.
    • AsyncHBaseSink (org.apache.flume.sink.hbase.AsyncHBaseSink) – It has improved performance than HBase sink as it can simply make non-blocking calls to HBase.

Working of the HBaseSink

• • • In HBaseSink, a Flume Event is changed over into HBase Increments or Puts. Serializer executes the HBaseEventSerializer which is then instantiated when the sink begins.
    • For each and every event, sink calls the initialize method in the serializer which then translates the Flume Event into HBase increments and puts to be sent to HBase cluster.

Working of the AsyncHBaseSink

• • This sink executes the AsyncHBaseEventSerializer. The initialize method is called just once by the sink when it begins.
  • The sink invokes the setEvent method and then makes calls to the getIncrements and getActions methods just similar to HBase sink. When the sink pause, the cleanUp method is called by the serializer.

• Consolidation in flume its collect data from different sources even it’s different flume Agents.
• Flume source can collect all data flow from different sources and flows through channel and sink. Finally send this data to HDFS or target destination.

• Flume can process streaming data. So if it is begun once, there is no stop or end to the process. Asynchronously it can flows information from source to HDFS by agent.
• First of all agent must know person components how they are associated to load information. So configuration is trigger to load the streaming data.
• For example consumer key, consumer secret access Token and access Token Secret are key factor to download data.
• To configure Flume, we have to modify three files namely, flume-env.sh, flumeconf.properties, and bash.rc.
• Setting the Path/Classpath, in the .bashrc file, set the home folder then the path, and the classpath for Flume as given below,

• In case you configure the apache flume you have the files are flume-conf.properties.template,flume-env.sh.template,flume-env.ps1.template, and log4j.properties.

• • Channel selectors control and dividing the events and allocating a specific channel. They are default or replicated channel selectors.

• Replicated channel selectors can duplicate the information in multiple or all channels. Multiplexing channel selectors used to divide and aggregate the data based on the events header data.
• It means based on sinks destination, the event aggregate into the specific sink.
• Example: One sink associated with hadoop, another with S3 another with Hbase, at that time, Multiplexing channel selectors can divide the events and flow to the specific sink.

• Overview and Architecture, a source can write at least with one or more channels.
• This is reason the property is plural channels instead of channel.
• There are two ways multiple channels can be handled. The event can be written to every channel or to only one channel, based on some Flume header value.
• The internal mechanism for this in Flume is known as channel selector.
• The selector for any channel can be determined using the selector.type property.

• Apache Flume offers interceptors as a method of modifying records as they have a Flume channel.
• Interceptors are used to filter the events between source and channel, channel and sink. These channels will filter un-necessary or targeted log files.

• Interceptors are a part of Flume’s extensibility model. They permit events to be inspected as they pass between a source and a channel, and also the developer is liberated to modify or drop events as needed. Interceptors in the chain along to create a pipeline process.

• Apache Flume is a reliable, distributed and accessible service for efficiently aggregating, collecting, and moving huge amounts of log data.
• It has an easy and flexible design based on streaming data flows. It’s robust and fault tolerant with tunable reliability mechanisms and recovery mechanisms.
• It uses an easy extensible data model that enables for on-line analytic application.

1. 1. Source
   2. Channel
   3. Sinks

• Sources define where data comes from, e.g. a file, a message queue (Kafka,JMS).
• Channels are pipes connecting sources with sinks.
• Sinks are the destination of the data pipelined from sources.