big data system design - Wikitechy https://www.wikitechy.com/interview-questions/tag/big-data-system-design/ Interview Questions Tue, 14 Sep 2021 09:51:21 +0000 en-US hourly 1 https://wordpress.org/?v=6.9.4 https://www.wikitechy.com/interview-questions/wp-content/uploads/2025/10/cropped-wikitechy-icon-32x32.png big data system design - Wikitechy https://www.wikitechy.com/interview-questions/tag/big-data-system-design/ 32 32 What are the steps to deploy a big data solution ? https://www.wikitechy.com/interview-questions/big-data/what-are-the-steps-to-deploy-a-big-data-solution/ https://www.wikitechy.com/interview-questions/big-data/what-are-the-steps-to-deploy-a-big-data-solution/#respond Mon, 12 Jul 2021 16:49:17 +0000 https://www.wikitechy.com/interview-questions/?p=262

What are the steps to deploy a big data solution ?

    • There are three steps to deploy a Big Data Solution
Deploying Big Data solution

Data Ingestion

  • The first step for deploying a big data solution is the data ingestion i.e. extraction of data from various sources.
  • The data source may be a CRM like Salesforce, Enterprise Resource Planning System like SAP, RDBMS like MySQL or any other log files, documents, social media feeds etc.
  • The data can be ingested either through batch jobs or real-time streaming. The extracted data is then stored in HDFS.
Data Ingestion in Big Data

Data Storage

  • After the data ingestion, the next step is to store the extracted data. The data either be stored in HDFS or NoSQL database (i.e. HBase).
  • The HDFS storage works well for sequential access whereas HBase for random read/write access.
Data Storage

Data Processing

  • The final step in deploying a big data solution is the data processing.
  • The data is processed through one of the processing frameworks like Spark, MapReduce, Pig, etc.
Data Processing in Big Data
]]> https://www.wikitechy.com/interview-questions/big-data/what-are-the-steps-to-deploy-a-big-data-solution/feed/ 0