[Solved-2 Solutions] Pig non-aggregated warnings output location ?
FIELD_DISCARDED_TYPE_CONVERSION_FAILEDwarnings, but its difficult to make individual warnings printed anywhere. Disabling aggregation via
aggregate.warnings=falseswitch removes the summary messages, BUT it does remove the actual warning too.
- There's nothing written in the pig's log for this run, AND there's no place you can locate the logs with the individual warnings. Is there any help?
- Hadoop job logs are recorded locally on each compute node. Therefore we first need to setup the hadoop cluster manager to collect the logfiles onto the distributed files system so that we can analyse them. If weuse Hadoop-on-demand .
- We need to specify the following thing:
After we have the logs on HDFS you can run a simple PIG query to find the offending conversion. Something like the following should do the trick:
- It's really complicated to find which data or value is causing issue, but at least we can find which column is creating this issue. Once we find the column we can use Dynamic Invoker which may help in type conversion.