Hot Swapping of Tables in Hive

Here are the steps to hot swap a table in Hive when you manually load data in them. Hot Swapping the table allows you to refresh an entire table without any downtime thus making your table always available for querying. With this approach you Lock the table but it is only for a fraction of a second. This is useful when you want to change the definition of a table or reload the whole table without downtime. Hive 11 allows you to maintain lock on a table which you can later drop and re-create.

Here are the steps for the swap:

  1. Store the data in a staging location in HDFS
  2. Get EXCLUSIVE LOCK on the Hive table (using LOCK TABLE tablename) - so queries have to wait to until we finish the refresh
  3. Delete the underlying Hive folder (e.g. /user/hive/warehouse/user.db/tablename). We have to delete first because of https://issues.apache.org/jira/browse/HDFS-4142 and https://issues.apache.org/jira/browse/HADOOP-9382)
  4. Move the staging folder to the hive directory. (e.g. mv /user/staging/tablename /user/hive/warehouse/user/db/tablename)
  5. Release LOCK on the table
 
35
Kudos
 
35
Kudos

Now read this

Two Very Useful Hive CLI settings

It is very helpful to set these in your .hiverc file. The hive cli reads from the .hiverc file in your home directory to override defaults. Two of the settings I find very important is set hive.cli.print.header=true; set... Continue →