• • Sqoop provides an incremental import mode which can be used to retrieve only rows newer than some previously-imported set of rows.

• Sqoop supports two types of incremental imports: append and lastmodified. You can use the --incremental argument to specify the type of incremental import to perform.
• You must specify append mode when importing a table where new rows are frequently being inserted with increasing row id values. You can specify the column containing the row’s id with --check-column. Sqoop imports rows where the verify column have a value greater than the single specified with --last-value.
• An alternating table update strategy supports by Sqoop is known lastmodified mode. You must use this while rows of the source table might be updated, and then each such update will set the value of a last-modified column to the present timestamp. Rows where have the check column holds a timestamp more and more new than the timestamp specified with --last-value are imported.

• Commodity hardware is a low-cost system identified by less-availability and low-quality.
• The commodity hardware comprises of RAM as it performs a number of services that require RAM for the execution.
• One doesn’t require high-end hardware configuration or supercomputers to run Hadoop, it can be run on any commodity hardware.

• • Linear and modular scalability.
  • Strictly consistent reads and writes.
  • Automatic and configurable sharding of tables.
  • Automatic failover support between RegionServers.

• • Convenient base classes for backing Hadoop MapReduce jobs with Apache HBase tables.
  • Easy to use Java API for client access.
  • Block cache and Bloom Filters for real-time queries.
  • Query predicate push down via server side Filters.
  • Thrift gateway and an REST-ful Web service that supports XML, Protobuf, and binary data encoding options.

How to use Apache HBase REST Interface

• Extensible JRuby-based (JIRB) shell.
• Support for exporting metrics via the Hadoop metrics subsystem to files or Ganglia; or via JMX.

• Hive is a datawarehousing package built on the top of Hadoop. It is mainly used for data analysis. It generally target towards users already comfortable with Structured Query Language (SQL).
• It is similar to SQL and called Hive Query Language (HQL).
• Hive manages and queries structured data. Moreover, hive abstracts complexity of Hadoop. It does not support
  • Not a full database.
  • Not a real time processing system.
  • Not SQL-92 compliant.
  • Does not provide row level insert, updates or deletes.
  • Doesn’t support transactions and limited sub-query support.
  • Query optimization in evolving stage.

• HBase is a column-oriented database management system that runs on top of Hadoop Distributed File System (HDFS).
• It is well suited for sparse data sets, which are common in many Big Data use cases.
• It is an opensource, distributed database developed by Apache software foundations.
• Initially, it was named Google Big Table, afterwards it was re-named as HBase and is primarily written in Java.
• It can store massive amount of data from terabytes to petabytes.
• It is built for low-latency operations and is used extensively for read and write operations.
• It stores large amount of data in the form of tables.

• Hadoop distributed file system (HDFS) uses a specific permissions model for files and directories. Following user levels are used in HDFS
  • Owner
  • Group
  • Others

• For each of the user mentioned above following permissions are applicable,
  • read (r)
  • write (w)
  • execute(x)

Above mentioned permissions work differently for files and directories.

• For files
  • read (r) permission – Reading a file.
  • write (w) permission – Writing a file.

• For directories.
  • The r permission lists the contents of a specific directory.
  • The w permission creates or deletes a directory.
  • The X permission accessing a child directory.

• NFS (Network File System) is one of the oldest and popular distributed file storage systems. Whereas HDFS (Hadoop Distributed File System) is the recently used and popular one to handle big data.

• Datasets in HDFS store as blocks in DataNodes the Hadoop cluster.
• During the execution of a MapReduce job the individual Mapper processes the blocks (Input Splits).
• If the data does not reside in the same node where the Mapper is executing the job, the data needs to be copied from the DataNode over the network to the mapper DataNode.

• Now if a MapReduce job has more than 100 Mapper and each Mapper tries to copy the data from other DataNode in the cluster simultaneously, it would cause serious network congestion which is a big performance issue of the overall system.
• Hence, data proximity to the computation is an effective and cost-effective solution which is technically termed as Data locality in Hadoop. It helps to increase the overall throughput of the system.

• Data local
  • In this type data and the mapper resides on the same node. This is the closest proximity of data and the most preferred scenario.

• Rack Local
  • In this type data and the mapper resides on the same node. This is the closest proximity of data and the most preferred scenario.
  • In this scenarios mapper and data reside on the same rack but on the different data nodes.

• Different Rack
  • In this scenario mapper and data reside on the different racks.

• This is the default mode and does not need any configuration.
• In this mode, all the following components of Hadoop uses local file system and runs on a single JVM
  • NameNode
  • DataNode
  • ResourceManager
  • NodeManager
• Suitable for running MapReduce programs during development.
• Has no DFS

• In this mode, all the master and slave Hadoop services are deployed and executed on a single node.

• In this mode, Hadoop master and slave services are deployed and executed on separate nodes.

• The main configuration parameters in “MapReduce” framework are:
  • Input location of Jobs in the distributed file system
  • Output location of Jobs in the distributed file system
  • The input format of data
  • The output format of data
  • The class which contains the map function
  • The class which contains the reduce function
  • JAR file which contains the mapper, reducer and the driver classes

• Hadoop is an open source framework that is meant for storage and processing of big data in a distributed manner.

• HDFS is the basic storage system of Hadoop.
• The large data files running on a cluster of commodity hardware are stored in HDFS.
• It can store data in a reliable manner even when hardware fails.

• Hadoop MapReduce
• YARN

• MapReduce is the Hadoop layer that is responsible for data processing. It writes an application to process unstructured and structured data stored in HDFS.
• It is responsible for the parallel processing of high volume of data by dividing data into independent tasks.
• The processing is done in two phases Map and Reduce.
• The Map is the first phase of processing that specifies complex logic code and the Reduce is the second phase of processing that specifies light-weight operations.

• The processing framework in Hadoop is YARN.
• It is used for resource management and provides multiple data processing engines i.e. data science, real-time streaming, and batch processing.

• Hadoop is an open source framework which means it is available free of cost.
• Also, the users are allowed to change the source code as per their requirements.

• Hadoop supports distributed processing of data i.e. faster processing.
• The data in Hadoop HDFS is stored in a distributed manner and MapReduce is responsible for the parallel processing of data.

• Hadoop is highly fault-tolerant. It creates three replicas for each block at different nodes, by default.
• This number can be changed according to the requirement. So, we can recover the data from another node if one node fails.
• The detection of node failure and recovery of data is done automatically.

• Hadoop stores data on the cluster in a reliable manner that is independent of machine.
• So, the data stored in Hadoop environment is not affected by the failure of the machine.

• Another important feature of Hadoop is the scalability. It is compatible with the other hardware and we can easily ass the new hardware to the nodes.

• Apache Hadoop is not very expensive as it runs on a cluster of commodity hardware.
• Hadoop also provides huge cost saving also as it is very easy to add more nodes on the fly here. So if requirement increases, then you can increase nodes as well without any downtime and without requiring much of pre-planning.

• No need of client to deal with distributed computing, the framework takes care of all the things. So this feature of Hadoop is easy to use.

• This one is a unique features of Hadoop that made it easily handle the Big Data. Hadoop works on data locality principle which states that move computation to data instead of data to computation.
• When a client submits the MapReduce algorithm, this algorithm is moved to data in the cluster rather than bringing data to the location where the algorithm is submitted and then processing it.

• The data stored in Hadoop is available to access even after the hardware failure. In case of hardware failure, the data can be accessed from another path.

• • The following steps need to execute to make the Hadoop cluster up and running :
    • Use the FsImage which is file system metadata replica to start a new NameNode.

• • Configure the DataNodes and also the clients to make them acknowledge the newly started NameNode.
  • Once the new NameNode completes loading the last checkpoint FsImage which has received enough block reports from the DataNodes, it will start to serve the client.
  • In case of large Hadoop clusters, the NameNode recovery process consumes a lot of time which turns out to be a more significant challenge in case of routine maintenance.

• Volume – Volume represents the volume i.e. amount of data that is growing at a high rate i.e. data volume in Petabytes.
• Velocity – Velocity is the rate at which data grows. Social media contributes a major role in the velocity of growing data.
• Variety – Variety refers to the different data types i.e. various data formats like text, audios, videos, etc.
• Variability – This refers to establishing if the contextualizing structure of the data stream is regular and dependable even in conditions of extreme unpredictability. It defines the need to get meaningful data considering all possible circumstances.
• Veracity – Veracity refers to the uncertainty of available data. Veracity arises due to the high volume of data that brings incompleteness and inconsistency.
• Value –Value refers to turning data into value. By turning accessed big data into values, businesses may generate revenue.

• Fsck stands for File System Check. This command is used by HDFS and also used to check inconsistencies and if there is any problem in the file.
• For example, if there are any missing blocks for a file, HDFS gets notified through this command.
• There are two modes used in fsck command interactive and non-interactive.
  • Interactive mode prompts the user each time it finds an error to determine whether or not the user wants to fix the error or continue without fixing.
  • Non-interactive mode automatically corrects errors without prompting the user.

• There are different arguments that can be passed with this command to emit different results.

• Big data analytics is the process of examining large data sets to uncover hidden patterns, unknown correlations, market trends, customer preferences and other useful business information.
• Hadoop is a framework to store and process big data. Hadoop specifically designed to provide distributed storage and parallel data processing that big data requires.

• High scalability – We can add any number of nodes, hence enhancing performance dramatically.
• High availability – In hadoop data is highly available despite hardware failure. If a machine or few hardware crashes, then we can access data from another path.
• Reliable – Data is reliably stored on the cluster despite of machine failure.
• Economic – Hadoop runs on a cluster of commodity hardware which is not very expensive.

• Hadoop is an open source project from Apache Software Foundation.
• It provides a software framework for distributing and running applications on clusters of servers that is inspired by Google’s Map-Reduce programming model as well as its file system(GFS).
• Hadoop was originally written for the nutch search engine project.
• Hadoop is open source framework written in Java. It efficiently processes large volumes of data on a cluster of commodity hardware.
• Hadoop can be setup on single machine , but the real power of Hadoop comes with a cluster of machines , it can be scaled from a single machine to thousands of nodes. Hadoop consists of two key parts,
  • Hadoop Distributes File System(HDFS)
  • Map-Reduce.

• HDFS is a highly fault tolerant, distributed, reliable, scalable file system for data storage.
• HDFS stores multiple copies of data on different nodes; a file is split up into blocks (Default 64 MB) and stored across multiple machines.
• Hadoop cluster typically has a single namenode and number of datanodes to form the HDFS cluster.

• Map-Reduce is a programming model designed for processing large volumes of data in parallel by dividing the work into a set of independent tasks.
• It is also a paradigm for distributed processing of large data set over a cluster of nodes.

• NameNode is the master node for processing metadata information for data blocks within the HDFS.

• DataNode is the node which acts as slave node to store the data, for processing and use by the NameNode.
• In addition to serving the client requests, the NameNode executes either of two following roles –
  • CheckpointNode.
  • BackupNode.

• It runs on a different host from the NameNode

• It is a read-only NameNode which contains file system metadata information excluding the block locations.

• This component receives processing requests and accordingly allocates to respective NodeManagers depending on processing needs.

• It executes tasks on each single Data Node.

• There are three steps to deploy a Big Data Solution

• The first step for deploying a big data solution is the data ingestion i.e. extraction of data from various sources.
• The data source may be a CRM like Salesforce, Enterprise Resource Planning System like SAP, RDBMS like MySQL or any other log files, documents, social media feeds etc.
• The data can be ingested either through batch jobs or real-time streaming. The extracted data is then stored in HDFS.

• After the data ingestion, the next step is to store the extracted data. The data either be stored in HDFS or NoSQL database (i.e. HBase).
• The HDFS storage works well for sequential access whereas HBase for random read/write access.

• The final step in deploying a big data solution is the data processing.
• The data is processed through one of the processing frameworks like Spark, MapReduce, Pig, etc.