my medium
my githubWhy merge ?
I had a table with a lot of regions (almost a thousand), more than I wanted, and more than I should, according to the HBase book.
The max size of the HFile hbase.hregion.max.filesize was 10G.
I raised it to 20G and recompact the whole table, thinking I’m done. But nothing happened.
Why ? Because HBase does not merge regions automatically.
Compaction is used to merge Storefiles in a same HStore (one HStore per column family, per region).
A region that exists, is going to exist forever. Except if we delete it or merge it manually.
I then decided to merge some regions, first to give a try, and second to see the impact in HDFS, because I’m curious.
If you are wondering too, you’re in the good place, keep reading. It’s not complex.
I’ll first do a merge without raising hbase.hregion.max.filesize, to see what is happening.
Then I’ll raise the max and do another merge, and check the differences.
HBase version : 1.0.0-cdh5.4.7 Hadoop Version : 2.6.0-cdh5.4.7
Merge still at 10G max
First, you need to find 2 consecutive regions to merge together.
The consecutivity is important, you can merge region that are not consecutive but it’s not recommanded (overlapping).
e.g. : If you have 2 regions which start/end keys are 0-9 and 9-A, you want to create a region which start/end keys are 0-A.
In HDFS, there is no order, it’s all guids. To know to what they correspond, one way is to go to the HBase admin and select the table.
That will display every of its regions name, uuid, start/end keys.
http://hadoopmaster:60010/table.jsp?name=my_table # or :16010 if recent
A region name is something like :
my_table,0115d0b6f99f58a34...2a9e72781c7,1440840915183.fbb00c100422d1cc0f9b7e39d6c6bd91. # signification : [table],[start key],[timestamp].[encoded ID]
The encoded ID is what we are interested into. This is the folder in HDFS (/hbase/data/default/my_table/fbb00c100422d1cc0f9b7e39d6c6bd91) where the data of this region are.
Let’s merge it with its following one.
hbase> merge_region 'fbb00c100422d1cc0f9b7e39d6c6bd91', ‘a12acd303c0b7e512c8926666c5f02eb'
That creates a new region 65bd... that contains a HFile which size is growing slowly, as we can see in HDFS :
(here is a diff from before and after the merge_region)
> 0 2015-12-23 12:24 .../my_table/65bd82b5477fcc2090804c351d89700a > 226 2015-12-23 12:24 .../my_table/65bd82b5477fcc2090804c351d89700a/.regioninfo > 0 2015-12-23 12:24 .../my_table/65bd82b5477fcc2090804c351d89700a/.tmp > 2684354560 2015-12-23 12:24 .../my_table/65bd82b5477fcc2090804c351d89700a/.tmp/752530e58ae8478d812696b066edcc9f > 0 2015-12-23 12:24 .../my_table/65bd82b5477fcc2090804c351d89700a/recovered.edits > 0 2015-12-23 12:24 .../my_table/65bd82b5477fcc2090804c351d89700a/recovered.edits/2206186528.seqid > 0 2015-12-23 12:24 .../my_table/65bd82b5477fcc2090804c351d89700a/t > 109 2015-12-23 12:24 .../my_table/65bd82b5477fcc2090804c351d89700a/t/ccd883e710664f1fbf605590deaf2868.a12acd303c0b7e512c8926666c5f02eb > 109 2015-12-23 12:24 .../my_table/65bd82b5477fcc2090804c351d89700a/t/e17b4ea9b9fa47c1839999426ef9ffe7.fbb00c100422d1cc0f9b7e39d6c6bd91 < 0 2015-12-23 12:13 .../my_table/a12acd303c0b7e512c8926666c5f02eb/recovered.edits < 0 2015-12-23 12:13 .../my_table/a12acd303c0b7e512c8926666c5f02eb/recovered.edits/2198106631.seqid --- > 0 2015-12-23 12:24 .../my_table/a12acd303c0b7e512c8926666c5f02eb/recovered.edits > 0 2015-12-23 12:24 .../my_table/a12acd303c0b7e512c8926666c5f02eb/recovered.edits/2198106637.seqid < 0 2015-12-23 11:45 .../my_table/fbb00c100422d1cc0f9b7e39d6c6bd91 --- > 0 2015-12-23 12:24 .../my_table/fbb00c100422d1cc0f9b7e39d6c6bd91 > 0 2015-12-23 12:24 .../my_table/fbb00c100422d1cc0f9b7e39d6c6bd91/.merges < 0 2015-12-23 12:13 .../my_table/fbb00c100422d1cc0f9b7e39d6c6bd91/recovered.edits < 0 2015-12-23 12:13 .../my_table/fbb00c100422d1cc0f9b7e39d6c6bd91/recovered.edits/2206186546.seqid --- > 0 2015-12-23 12:24 .../my_table/fbb00c100422d1cc0f9b7e39d6c6bd91/recovered.edits > 0 2015-12-23 12:24 .../my_table/fbb00c100422d1cc0f9b7e39d6c6bd91/recovered.edits/2206186549.seqid
What we can see :
– new region in the folder 65b... with a HFile in .tmp (2.7GB, growing) and a .regioninfo (very important, that’s the metadata to identify what is this region)
– new empty folder .merges in one of the region we are merging
– recovered.edits folders. Don’t mind them, I won’t display them anymore. For more info, check this nice Cloudera blog post to know more about it.
After a few minutes, it was done, the HFile grew up to 17GB (which was over the limit of 10GB).
HBase started the reverse process : it split the big region I just made ! :-(
> 0 2015-12-23 13:05 .../my_table/2c142664dc0929d7c6cc5fa6fe3b4e40 > 226 2015-12-23 13:05 .../my_table/2c142664dc0929d7c6cc5fa6fe3b4e40/.regioninfo > 0 2015-12-23 13:05 .../my_table/2c142664dc0929d7c6cc5fa6fe3b4e40/t > 109 2015-12-23 13:05 .../my_table/2c142664dc0929d7c6cc5fa6fe3b4e40/t/752530e58ae8478d812696b066edcc9f.65bd82b5477fcc2090804c351d89700a > 0 2015-12-23 13:05 .../my_table/65bd82b5477fcc2090804c351d89700a > 226 2015-12-23 12:24 .../my_table/65bd82b5477fcc2090804c351d89700a/.regioninfo > 0 2015-12-23 13:05 .../my_table/65bd82b5477fcc2090804c351d89700a/.splits > 0 2015-12-23 13:05 .../my_table/65bd82b5477fcc2090804c351d89700a/.tmp > 0 2015-12-23 13:05 .../my_table/65bd82b5477fcc2090804c351d89700a/t > 17860937303 2015-12-23 13:05 .../my_table/65bd82b5477fcc2090804c351d89700a/t/752530e58ae8478d812696b066edcc9f > 0 2015-12-23 13:05 .../my_table/743bfa035be56bf412d00803abe433b8 > 226 2015-12-23 13:05 .../my_table/743bfa035be56bf412d00803abe433b8/.regioninfo > 0 2015-12-23 13:05 .../my_table/743bfa035be56bf412d00803abe433b8/.tmp > 134217728 2015-12-23 13:05 .../my_table/743bfa035be56bf412d00803abe433b8/.tmp/e377603958894f8ca1ec598112b95bf4 > 0 2015-12-23 13:05 .../my_table/743bfa035be56bf412d00803abe433b8/t > 109 2015-12-23 13:05 .../my_table/743bfa035be56bf412d00803abe433b8/t/752530e58ae8478d812696b066edcc9f.65bd82b5477fcc2090804c351d89700a < 0 2015-12-23 11:45 .../my_table/a12acd303c0b7e512c8926666c5f02eb < 0 2015-12-23 11:45 .../my_table/fbb00c100422d1cc0f9b7e39d6c6bd91
– the two old regions are removed (a12... and fbb...)
– the split region has a .splits folder
– 2 new regions appeared : 2c1... and 743...
– only one of these 2 regions have a HFile that is slowly growing (meaning: sequential process)
Meanwhile, in the logs…
// 2 new regions from a SPLIT
2015-12-23 13:05:32,817 INFO org.apache.hadoop.hbase.master.RegionStates: Transition null to {2c142664dc0929d7c6cc5fa6fe3b4e40 state=SPLITTING_NEW, ts=1450872332817, server=hadoopslave04,60020,1450869198826}
2015-12-23 13:05:32,817 INFO org.apache.hadoop.hbase.master.RegionStates: Transition null to {743bfa035be56bf412d00803abe433b8 state=SPLITTING_NEW, ts=1450872332817, server=hadoopslave04,60020,1450869198826}
// the region we are splitting was OPEN
// it goes to SPLITTING then SPLIT, and is set offline for the time being
2015-12-23 13:05:32,817 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {65bd82b5477fcc2090804c351d89700a state=OPEN, ts=1450869854560, server=hadoopslave04,60020,1450869198826} to {65bd82b5477fcc2090804c351d89700a state=SPLITTING, ts=1450872332817, server=hadoopslave04,60020,1450869198826}
2015-12-23 13:05:34,767 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {65bd82b5477fcc2090804c351d89700a state=SPLITTING, ts=1450872334767, server=hadoopslave04,60020,1450869198826} to {65bd82b5477fcc2090804c351d89700a state=SPLIT, ts=1450872334767, server=hadoopslave04,60020,1450869198826}
2015-12-23 13:05:34,767 INFO org.apache.hadoop.hbase.master.RegionStates: Offlined 65bd82b5477fcc2090804c351d89700a from hadoopslave04,60020,1450869198826
// both 2 new regions switch from SPLITTING_NEW to OPEN
2015-12-23 13:05:34,767 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {2c142664dc0929d7c6cc5fa6fe3b4e40 state=SPLITTING_NEW, ts=1450872334767, server=hadoopslave04,60020,1450869198826} to {2c142664dc0929d7c6cc5fa6fe3b4e40 state=OPEN, ts=1450872334767, server=hadoopslave04,60020,1450869198826}
2015-12-23 13:05:34,767 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {743bfa035be56bf412d00803abe433b8 state=SPLITTING_NEW, ts=1450872334767, server=hadoopslave04,60020,1450869198826} to {743bfa035be56bf412d00803abe433b8 state=OPEN, ts=1450872334767, server=hadoopslave04,60020,1450869198826}
// daughter a and b = new regions with start keys; the parent being the split region
2015-12-23 13:05:34,873 INFO org.apache.hadoop.hbase.master.AssignmentManager: Handled SPLIT event; parent=my_table,fe7f...,1450869853820.65bd82b5477fcc2090804c351d89700a., daughter a=my_table,fe7f...,1450872332556.2c142664dc0929d7c6cc5fa6fe3b4e40., daughter b=my_table,feff7...,1450872332556.743bfa035be56bf412d00803abe433b8., on hadoopslave04,60020,1450869198826
// then the split region saw it reference in the metadata (zookeeper) removed
2015-12-23 13:08:28,965 INFO org.apache.hadoop.hbase.MetaTableAccessor: Deleted references in merged region my_table,fe7f...,1450869853820.65bd82b5477fcc2090804c351d89700a., qualifier=mergeA and qualifier=mergeB
Back to HDFS
After a while, HBase is done with the daughter b region 743b..., and starts creating the daughter a region 2c14....
> 0 2015-12-23 13:25 .../my_table/2c142664dc0929d7c6cc5fa6fe3b4e40 > 226 2015-12-23 13:05 .../my_table/2c142664dc0929d7c6cc5fa6fe3b4e40/.regioninfo > 0 2015-12-23 13:41 .../my_table/2c142664dc0929d7c6cc5fa6fe3b4e40/.tmp > 0 2015-12-23 13:41 .../my_table/2c142664dc0929d7c6cc5fa6fe3b4e40/t > 8732040437 2015-12-23 13:41 .../my_table/2c142664dc0929d7c6cc5fa6fe3b4e40/t/2388513b0d55429888478924914af494 > 0 2015-12-23 13:05 .../my_table/65bd82b5477fcc2090804c351d89700a > 226 2015-12-23 12:24 .../my_table/65bd82b5477fcc2090804c351d89700a/.regioninfo > 0 2015-12-23 13:05 .../my_table/65bd82b5477fcc2090804c351d89700a/.splits > 0 2015-12-23 13:05 .../my_table/65bd82b5477fcc2090804c351d89700a/.tmp > 0 2015-12-23 13:05 .../my_table/65bd82b5477fcc2090804c351d89700a/t > 17860937303 2015-12-23 13:05 .../my_table/65bd82b5477fcc2090804c351d89700a/t/752530e58ae8478d812696b066edcc9f > 0 2015-12-23 13:05 .../my_table/743bfa035be56bf412d00803abe433b8 > 226 2015-12-23 13:05 .../my_table/743bfa035be56bf412d00803abe433b8/.regioninfo > 0 2015-12-23 13:25 .../my_table/743bfa035be56bf412d00803abe433b8/.tmp > 0 2015-12-23 13:25 .../my_table/743bfa035be56bf412d00803abe433b8/t > 8733203481 2015-12-23 13:25 .../my_table/743bfa035be56bf412d00803abe433b8/t/e377603958894f8ca1ec598112b95bf4
It’s done. The region has been successfully split.
After a few minutes, the big region 65bd... will be removed automatically.
2015-12-23 13:43:28,908 INFO org.apache.hadoop.hbase.MetaTableAccessor: Deleted my_table,fe7f...,1450869853820.65bd82b5477fcc2090804c351d89700a. 2015-12-23 13:43:28,908 INFO org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 722 catalog row(s), gc'd 0 unreferenced merged region(s) and 1 unreferenced parent region(s)
Note: if we do the difference between the sum of the daughter region sizes and the big one, we get a delta of +395MB (the single HFile is bigger).
We’ve successfully merged 2 regions into one, that was automatically split into two. Hurray !
Raising hbase.hregion.max.filesize to avoid splitting
Now, let’s change hbase.hregion.max.filesize in Cloudera to 20G and merge again to get a big one, without split.
We apply the same process than before and merge manually the 2 regions we got previously 2c14... and 743b....
That creates a new region 1e64... which size is surprisingly lesser than our previous merge (we only get a delta of 212KB) that is not going to be split.
$ hdfs dfs -ls -R /hbase/data/default/my_table/1e64aa6f3f5cf067f6d5339230ef6db7 -rw-r--r-- 3 hbase hbase 226 2015-12-23 13:45 /hbase/data/default/my_table/1e64aa6f3f5cf067f6d5339230ef6db7/.regioninfo drwxr-xr-x - hbase hbase 0 2015-12-23 14:12 /hbase/data/default/my_table/1e64aa6f3f5cf067f6d5339230ef6db7/.tmp drwxr-xr-x - hbase hbase 0 2015-12-23 13:48 /hbase/data/default/my_table/1e64aa6f3f5cf067f6d5339230ef6db7/recovered.edits -rw-r--r-- 3 hbase hbase 0 2015-12-23 13:48 /hbase/data/default/my_table/1e64aa6f3f5cf067f6d5339230ef6db7/recovered.edits/2206186536.seqid drwxr-xr-x - hbase hbase 0 2015-12-23 14:12 /hbase/data/default/my_table/1e64aa6f3f5cf067f6d5339230ef6db7/t -rw-r--r-- 3 hbase hbase 17465031518 2015-12-23 14:12 /hbase/data/default/my_table/1e64aa6f3f5cf067f6d5339230ef6db7/t/d1109c52de404b0c9d07e2e9c7fdeb5e
So that worked, I have one less region ! Let’s continue why the thousands more I have.
Why should I know this ?
Knowing what’s going on in HDFS with HBase is important when you are facing issues and errors in HBase table structure.
To know if you have that kind of issues, just give a try with :
$ hbase hbck my_table
If you see some ERRORS such as :
– No HDFS region dir found
– Region not deployed on any region server.
– Region found in META, but not in HDFS or deployed on any region server.
– First region should start with an empty key. You need to create a new region and regioninfo in HDFS to plug the hole.
– You need to create a new .regioninfo and region dir in hdfs to plug the hole.
– ERROR: Found lingering reference file hdfs://…
That’s going to be a bit useful. (and maybe complicated for you, that depends of the problem!)
That can happen quite easily unfortunately (I successfully ran into those issue just with a merge_region, not sure why exactly).
