Repository navigation
HBASE-30463 HMaster fails to initialize when the procedure store contains more than one ServerCrashProcedure for the same server - #8765
Merged
Conversation
…ains more than one ServerCrashProcedure for the same server Merge duplicate SCPs per server with Math::min instead of throwing on the duplicate key. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
@petersomogyi I opened a new PR for https://issues.apache.org/jira/projects/HBASE/issues/HBASE-30463 |
petersomogyi
approved these changes
Oct 13, 2026
petersomogyi
pushed a commit
that referenced
this pull request
Oct 14, 2026
…ains more than one ServerCrashProcedure for the same server (#8765) Merge duplicate SCPs per server with Math::min instead of throwing on the duplicate key. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Peter Somogyi <psomogyi@apache.org> (cherry picked from commit 27396d6) (cherry picked from commit c535cd7)
petersomogyi
pushed a commit
that referenced
this pull request
Oct 14, 2026
…ains more than one ServerCrashProcedure for the same server (#8765) Merge duplicate SCPs per server with Math::min instead of throwing on the duplicate key. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Peter Somogyi <psomogyi@apache.org> (cherry picked from commit 27396d6)
petersomogyi
pushed a commit
that referenced
this pull request
Oct 14, 2026
…ains more than one ServerCrashProcedure for the same server (#8765) Merge duplicate SCPs per server with Math::min instead of throwing on the duplicate key. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Peter Somogyi <psomogyi@apache.org> (cherry picked from commit 27396d6)
petersomogyi
pushed a commit
that referenced
this pull request
Oct 14, 2026
…ains more than one ServerCrashProcedure for the same server (#8765) Merge duplicate SCPs per server with Math::min instead of throwing on the duplicate key. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Peter Somogyi <psomogyi@apache.org> (cherry picked from commit c535cd7)
petersomogyi
pushed a commit
that referenced
this pull request
Oct 14, 2026
…ains more than one ServerCrashProcedure for the same server (#8765) Merge duplicate SCPs per server with Math::min instead of throwing on the duplicate key. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Peter Somogyi <psomogyi@apache.org> (cherry picked from commit 27396d6)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/HBASE-30463
Since HBASE-28158, HMaster#finishActiveMasterInitialization collects the unfinished
ServerCrashProcedures with Collectors.toMap(ServerCrashProcedure::getServerName, Procedure::getSubmittedTime) and no merge function. If the procedure store holds two
unfinished SCPs for the same ServerName, the collector throws
IllegalStateException: Duplicate key ... and the master aborts. It aborts again on
every restart, because the same store is reloaded. Before HBASE-28158 these procedures
were collected into a Set, which tolerated duplicates.
Duplicate SCPs can be persisted because the submission guards rely on in-memory
DeadServer state. DeadServer#cleanPreviousInstance removes the old server name while
its SCP is still running, as soon as a new instance on the same host and port reports in.
The fix adds Math::min as the merge function. That keeps the earliest submit time as
the crash time, matching DeadServer#putIfAbsent. Both SCPs are still loaded and run
one after the other, because they take the same exclusive server lock.
TestMasterStartupWithDuplicateSCPs holds the server lock so that two SCPs for the same
server stay unfinished in the store. It then restarts the master and verifies that the
master initializes and both procedures complete.
Affects 2.5.14+, 2.6.5+, 2.7.0 and 3.0.0. This patch is on master
🤖 Generated with Claude Code