hadoop framework - Wikitechy https://www.wikitechy.com/interview-questions/tag/hadoop-framework/ Interview Questions Wed, 22 Sep 2021 05:53:05 +0000 en-US hourly 1 https://wordpress.org/?v=6.9 https://www.wikitechy.com/interview-questions/wp-content/uploads/2025/10/cropped-wikitechy-icon-32x32.png hadoop framework - Wikitechy https://www.wikitechy.com/interview-questions/tag/hadoop-framework/ 32 32 Difference between nfs and hdfs ? https://www.wikitechy.com/interview-questions/big-data/difference-between-nfs-and-hdfs/ https://www.wikitechy.com/interview-questions/big-data/difference-between-nfs-and-hdfs/#respond Mon, 12 Jul 2021 18:26:28 +0000 https://www.wikitechy.com/interview-questions/?p=289

Difference between nfs and hdfs ?

  • NFS (Network File System) is one of the oldest and popular distributed file storage systems. Whereas HDFS (Hadoop Distributed File System) is the recently used and popular one to handle big data.
Difference between nfs and hdfs
CRITERIA NFS HDFS
Data Size Support NFS can store and process small
amount of data.		 HDFS is mainly use to store and process big data.
Data Storage Data is stored on a single dedicated
hardware.		 The data blocks are distributed on the local drives of hardware.
Reliability No reliability, data is not available
in the case of machine failure.		 Data is stored reliably, data is available even after machine failure.
Data Redundancy NFS runs on single machine,
no chances of data redundancy.		 HDFS runs on a cluster of different machines, data redundancy may occur due to replication protocol.
Target Users Workgroup. Larger than AFS.
Domain Single Domain. Multi Domain.
Client Server Trust Client identity is trusted by default. Client identity is what os tells. No Kerberos Auth.
Compatability with O/S Same System calls as of O/S. Different Calls.Mainly used for non interactive programs.
]]> https://www.wikitechy.com/interview-questions/big-data/difference-between-nfs-and-hdfs/feed/ 0 Why do we need Data Locality in Hadoop ? https://www.wikitechy.com/interview-questions/big-data/why-do-we-need-data-locality-in-hadoop/ https://www.wikitechy.com/interview-questions/big-data/why-do-we-need-data-locality-in-hadoop/#respond Mon, 12 Jul 2021 18:21:39 +0000 https://www.wikitechy.com/interview-questions/?p=287

Why do we need Data Locality in Hadoop ?

Data Locality in Hadoop
  • Datasets in HDFS store as blocks in DataNodes the Hadoop cluster.
  • During the execution of a MapReduce job the individual Mapper processes the blocks (Input Splits).
  • If the data does not reside in the same node where the Mapper is executing the job, the data needs to be copied from the DataNode over the network to the mapper DataNode.
Datasets in HDFS - Data Locality in Hadoop
  • Now if a MapReduce job has more than 100 Mapper and each Mapper tries to copy the data from other DataNode in the cluster simultaneously, it would cause serious network congestion which is a big performance issue of the overall system.
  • Hence, data proximity to the computation is an effective and cost-effective solution which is technically termed as Data locality in Hadoop. It helps to increase the overall throughput of the system.

Types of data locality

  • Data local
    • In this type data and the mapper resides on the same node. This is the closest proximity of data and the most preferred scenario.
  • Rack Local
    • In this type data and the mapper resides on the same node. This is the closest proximity of data and the most preferred scenario.
    • In this scenarios mapper and data reside on the same rack but on the different data nodes.
  • Different Rack
    • In this scenario mapper and data reside on the different racks.
Types of data locality
]]> https://www.wikitechy.com/interview-questions/big-data/why-do-we-need-data-locality-in-hadoop/feed/ 0 What are the core components of Hadoop ? https://www.wikitechy.com/interview-questions/big-data/what-are-the-core-components-of-hadoop/ https://www.wikitechy.com/interview-questions/big-data/what-are-the-core-components-of-hadoop/#respond Mon, 12 Jul 2021 18:01:53 +0000 https://www.wikitechy.com/interview-questions/?p=281

What are the core components of Hadoop ?

  • Hadoop is an open source framework that is meant for storage and processing of big data in a distributed manner.
What are the core components of Hadoop

HDFS (Hadoop Distributed File System)

  • HDFS is the basic storage system of Hadoop.
  • The large data files running on a cluster of commodity hardware are stored in HDFS.
  • It can store data in a reliable manner even when hardware fails.
HDFS (Hadoop Distributed File System)

Core Components of Hadoop

  • Hadoop MapReduce
  • YARN

Hadoop MapReduce

  • MapReduce is the Hadoop layer that is responsible for data processing. It writes an application to process unstructured and structured data stored in HDFS.
  • It is responsible for the parallel processing of high volume of data by dividing data into independent tasks.
  • The processing is done in two phases Map and Reduce.
  • The Map is the first phase of processing that specifies complex logic code and the Reduce is the second phase of processing that specifies light-weight operations.
Hadoop MapReduce

YARN

  • The processing framework in Hadoop is YARN.
  • It is used for resource management and provides multiple data processing engines i.e. data science, real-time streaming, and batch processing.
YARN
]]> https://www.wikitechy.com/interview-questions/big-data/what-are-the-core-components-of-hadoop/feed/ 0