Create a Compressed RC file table in HIVE

Here are the config parameters to set in the hive client when you want to create a compressed RC file table in HIVE. Note: RC files can only be created when the data is already in HDFS. Unfortunately, I haven’t figured out a way for it to work with LOAD DATA LOCAL……

SET hive.exec.compress.output=true;
SET mapred.max.split.size=256000000;
SET mapred.output.compression.type=BLOCK;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
 
29
Kudos
 
29
Kudos

Now read this

Streaming data to Hadoop using Unix Pipes? Use Pipefail

If you pipe the output of a statement to hadoop streaming you must know about the unix pipefail option. To demonstrate what it does, try this out in your commandline: $> true | false $> echo $? 1 $> false | true $> echo $? 0... Continue →