flume interview questions - Wikitechy

What are the channel types in Flume and Which channel type is faster ?

Editor — Sun, 11 Jul 2021 16:10:38 +0000

Channels in Flume

It stores events; events are delivered to the channel by means of sources operating within the agent.
An event remains in the channel until a sink removes it for further transport.

There are three different built in channels in Flume –

MEMORY Channel – Events are study from the source into memory and passed to the sink.
JDBC Channel – It stores the events in an embedded Derby database.
FILE Channel – It writes the contents to a file on the file system after reading the event from a source. The file is erased only after the contents are successfully delivered to the sink.

MEMORY Channel – It is the fastest channel among the three however has the risk of data loss. The channel that you choose totally depends on the nature of the big data application and the value of each event.

What are the steps in Flume configurations ?

Editor — Sun, 11 Jul 2021 16:00:29 +0000

Steps in Flume configurations

Flume can process streaming data. So if it is begun once, there is no stop or end to the process. Asynchronously it can flows information from source to HDFS by agent.
First of all agent must know person components how they are associated to load information. So configuration is trigger to load the streaming data.
For example consumer key, consumer secret access Token and access Token Secret are key factor to download data.
To configure Flume, we have to modify three files namely, flume-env.sh, flumeconf.properties, and bash.rc.
Setting the Path/Classpath, in the .bashrc file, set the home folder then the path, and the classpath for Flume as given below,

In case you configure the apache flume you have the files are flume-conf.properties.template,flume-env.sh.template,flume-env.ps1.template, and log4j.properties.

What is Channel Selectors ?

Editor — Sun, 11 Jul 2021 15:50:47 +0000

Channel Selectors

- Channel selectors control and dividing the events and allocating a specific channel. They are default or replicated channel selectors.

Replicated channel selectors can duplicate the information in multiple or all channels. Multiplexing channel selectors used to divide and aggregate the data based on the events header data.
It means based on sinks destination, the event aggregate into the specific sink.
Example: One sink associated with hadoop, another with S3 another with Hbase, at that time, Multiplexing channel selectors can divide the events and flow to the specific sink.

Learn Flume – Flume tutorial – what are channel selectors – Flume examples – Flume programs

Overview and Architecture, a source can write at least with one or more channels.
This is reason the property is plural channels instead of channel.
There are two ways multiple channels can be handled. The event can be written to every channel or to only one channel, based on some Flume header value.
The internal mechanism for this in Flume is known as channel selector.
The selector for any channel can be determined using the selector.type property.

All selector-specific properties start with the typical Source prefix: the agent name, keyword sources, and source name:

agent.sources.s1.selector.type=replicating