Process to perform an incremental data load in sqoop:
-
- Sqoop provides an incremental import mode which can be used to retrieve only rows newer than some previously-imported set of rows.
- Sqoop supports two types of incremental imports:
appendandlastmodified.You can use the--incrementalargument to specify the type of incremental import to perform. - You must specify
appendmode when importing a table where new rows are frequently being inserted with increasing row id values. You can specify the column containing the row’s id with--check-column.Sqoop imports rows where the verify column have a value greater than the single specified with--last-value. - An alternating table update strategy supports by Sqoop is known
lastmodifiedmode. You must use this while rows of the source table might be updated, and then each such update will set the value of a last-modified column to the present timestamp. Rows where have the check column holds a timestamp more and more new than the timestamp specified with--last-valueare imported.
