sqoop - sqoop Import New as well as Updated Data - lastmodified mode - append mode - apache sqoop - sqoop tutorial - sqoop hadoop
Use Sqoop to make incremental copy of a Oracle table to Hadoop
sqoop job \
--create alarms \
-- \
import \
--connect jdbc:oracle:thin:@devdb11-s.cern.ch:10121/devdb11_s.cern.ch \
--username hadoop_tutorial \
-P \
--num-mappers 1 \
--target-dir lemontest_alarms_i \
--table LEMONTEST.ALARMS \
--incremental append \
--check-column alarm_id \
--last-value 0 \Click "Copy code" button to copy into clipboard - By wikitechy - sqoop tutorial - team
sqoop job --exec alarms
Click "Copy code" button to copy into clipboard - By wikitechy - sqoop tutorial - team
sqoop import \
--connect jdbc:oracle:thin:@devdb11-s.cern.ch:10121/devdb11_s.cern.ch \
--username hadoop_tutorial \
-P \
--num-mappers 1 \
--table LEMONTEST.ALARMS \
--target-dir lemontest_alarms_i \
--incremental append \
--check-column alarm_id \
--last-value 47354 \
hdfs dfs -ls lemontest_alarms_i/Click "Copy code" button to copy into clipboard - By wikitechy - sqoop tutorial - team
sqoop Import New as well as Updated Data - lastmodified mode
- --check-column : A column name that should be checked for newly appended and updated data. date, time, datetime and timestamp are suitable data types for this column
- --last-value : The last value that successfully imported into Hadoop. All the newly added and updated data after this value will be imported.
sqoop import \
--connect jdbc:mysql://mysql.example.com/testdb \
--username sqoop \
--password sqoop \
--table employee \
--incremental lastmodified \
--check-column last_update_date \
--last-value "2015-10-20 06:00:01"Click "Copy code" button to copy into clipboard - By wikitechy - sqoop tutorial - team
Import New as well as Updated Data - Append Mode
- --check-column : A column name that should be checked for newly appended data. Integer would be a suitable data type for this column.
- --last-value : The last value that successfully imported into Hadoop. All the newly added data after this value will be imported.
sqoop import \
--connect jdbc:mysql://mysql.example.com/testdb \
--username sqoop \
--password sqoop \
--table employee \
--incremental append \
--check-column id \
--last-value 100