hadoop storage - Wikitechy https://www.wikitechy.com/interview-questions/tag/hadoop-storage/ Interview Questions Tue, 14 Sep 2021 09:47:12 +0000 en-US hourly 1 https://wordpress.org/?v=6.9 https://www.wikitechy.com/interview-questions/wp-content/uploads/2025/10/cropped-wikitechy-icon-32x32.png hadoop storage - Wikitechy https://www.wikitechy.com/interview-questions/tag/hadoop-storage/ 32 32 What are the core components of Hadoop ? https://www.wikitechy.com/interview-questions/big-data/what-are-the-core-components-of-hadoop/ https://www.wikitechy.com/interview-questions/big-data/what-are-the-core-components-of-hadoop/#respond Mon, 12 Jul 2021 18:01:53 +0000 https://www.wikitechy.com/interview-questions/?p=281

What are the core components of Hadoop ?

  • Hadoop is an open source framework that is meant for storage and processing of big data in a distributed manner.
What are the core components of Hadoop

HDFS (Hadoop Distributed File System)

  • HDFS is the basic storage system of Hadoop.
  • The large data files running on a cluster of commodity hardware are stored in HDFS.
  • It can store data in a reliable manner even when hardware fails.
HDFS (Hadoop Distributed File System)

Core Components of Hadoop

  • Hadoop MapReduce
  • YARN

Hadoop MapReduce

  • MapReduce is the Hadoop layer that is responsible for data processing. It writes an application to process unstructured and structured data stored in HDFS.
  • It is responsible for the parallel processing of high volume of data by dividing data into independent tasks.
  • The processing is done in two phases Map and Reduce.
  • The Map is the first phase of processing that specifies complex logic code and the Reduce is the second phase of processing that specifies light-weight operations.
Hadoop MapReduce

YARN

  • The processing framework in Hadoop is YARN.
  • It is used for resource management and provides multiple data processing engines i.e. data science, real-time streaming, and batch processing.
YARN
]]> https://www.wikitechy.com/interview-questions/big-data/what-are-the-core-components-of-hadoop/feed/ 0 Why Hadoop used for Big Data Analytics ? https://www.wikitechy.com/interview-questions/big-data/why-hadoop-used-for-big-data-analytics/ https://www.wikitechy.com/interview-questions/big-data/why-hadoop-used-for-big-data-analytics/#respond Mon, 12 Jul 2021 17:16:29 +0000 https://www.wikitechy.com/interview-questions/?p=269

Why Hadoop used for Big Data Analytics ?

  • Big data analytics is the process of examining large data sets to uncover hidden patterns, unknown correlations, market trends, customer preferences and other useful business information.
  • Hadoop is a framework to store and process big data. Hadoop specifically designed to provide distributed storage and parallel data processing that big data requires.

Hadoop is the best solution for storing and processing big data because:

Hadoop stores huge files as they are (raw) without specifying any schema.

  • High scalability – We can add any number of nodes, hence enhancing performance dramatically.
  • High availability – In hadoop data is highly available despite hardware failure. If a machine or few hardware crashes, then we can access data from another path.
  • Reliable – Data is reliably stored on the cluster despite of machine failure.
  • Economic – Hadoop runs on a cluster of commodity hardware which is not very expensive.

What is Hadoop ?

  • Hadoop is an open source project from Apache Software Foundation.
  • It provides a software framework for distributing and running applications on clusters of servers that is inspired by Google’s Map-Reduce programming model as well as its file system(GFS).
  • Hadoop was originally written for the nutch search engine project.
  • Hadoop is open source framework written in Java. It efficiently processes large volumes of data on a cluster of commodity hardware.
  • Hadoop can be setup on single machine , but the real power of Hadoop comes with a cluster of machines , it can be scaled from a single machine to thousands of nodes. Hadoop consists of two key parts,
    • Hadoop Distributes File System(HDFS)
    • Map-Reduce.
Hadoop Overview

Hadoop Distributed File System(HDFS)

  • HDFS is a highly fault tolerant, distributed, reliable, scalable file system for data storage.
  • HDFS stores multiple copies of data on different nodes; a file is split up into blocks (Default 64 MB) and stored across multiple machines.
  • Hadoop cluster typically has a single namenode and number of datanodes to form the HDFS cluster.

Map-Reduce

  • Map-Reduce is a programming model designed for processing large volumes of data in parallel by dividing the work into a set of independent tasks.
  • It is also a paradigm for distributed processing of large data set over a cluster of nodes.
]]> https://www.wikitechy.com/interview-questions/big-data/why-hadoop-used-for-big-data-analytics/feed/ 0