[Solved-1 Solution] Hadoop Pig - Removing csv header ?
What is hadoop ?
- Apache Hadoop is an open-source software framework used for distributed storage and processing of dataset of big data using the MapReduce programming model. It consists of computer clusters built from commodity hardware
- Hadoop File System (CSV) Use the Hadoop File System (CSV) tab to specify processing options for converting to HDFS CSV format and to enter file names for object-specific target files. The tab displays when you select the Hadoop File System - CSV format for the converted file.
Here the csv files have header in the first line. Loading them into pig create a mess on any subsequent functions (like SUM). Here to apply a filter on the loaded data to remove the rows containing the headers:
Is there is a way to tell pig not to load the first line of the csv, like an "as_header" boolean parameter to the load function. What would be a best practice?
- CSVExcelStorage loader support to skip the header row, so instead of PigStorage use CSVExcelStorage. Download piggybank.jar and try this option.
PigScript:(With SKIP_INPUT_HEADER option)