<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0" /> <base href="https://dev.majordodo.org/jira" />
<title>Message Title</title>
</head>
<body class="jira" style="color: #333; font-family: Arial, sans-serif; font-size: 14px; line-height: 1.429">
<table id="background-table" cellpadding="0" cellspacing="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; background-color: #f5f5f5; border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<!-- header here -->
<tr>
<td id="header-pattern-container" style="padding: 0px; border-collapse: collapse; padding: 10px 20px">
<table id="header-pattern" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td id="header-avatar-image-container" valign="top" style="padding: 0px; border-collapse: collapse; vertical-align: top; width: 32px; padding-right: 8px"> <img id="header-avatar-image" class="image_fix" src="cid:jira-generated-image-avatar-f8799499-a671-49ce-a2d2-c05130fa1992" height="32" width="32" border="0" style="border-radius: 3px; vertical-align: top" />
</td>
<td id="header-text-container" valign="middle" style="padding: 0px; border-collapse: collapse; vertical-align: middle; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px"> <a class="user-hover" rel="alessandro.luccaroni" id="email_alessandro.luccaroni" href="https://dev.majordodo.org/jira/secure/ViewProfile.jspa?name=alessandro.luccaroni" style="color:#3b73af;; color: #3b73af; text-decoration: none">Alessandro Luccaroni</a> <strong>created</strong> an issue
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="email-content-container" style="padding: 0px; border-collapse: collapse; padding: 0 20px">
<table id="email-content-table" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; border-spacing: 0; border-collapse: separate">
<tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<td class="email-content-rounded-top mobile-expand" style="padding: 0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 15px; background-color: #fff; border-left: 1px solid #ccc; border-top: 1px solid #ccc; border-right: 1px solid #ccc; border-bottom: 0; border-top-right-radius: 5px; border-top-left-radius: 5px; height: 10px; line-height: 10px; padding: 0 15px 0 16px; mso-line-height-rule: exactly">
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff">
<table class="page-title-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td class="page-title-pattern-first-line " style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; padding-top: 10px"> <a href="https://dev.majordodo.org/jira/browse/MAJ" style="color: #3b73af; text-decoration: none">Majordodo</a> / <a href="https://dev.majordodo.org/jira/browse/MAJ-60" style="color: #3b73af; text-decoration: none"><img src="cid:jira-generated-image-avatar-9ea8acdd-07b2-4988-bef0-d50b79eaee64" height="16" width="16" border="0" align="absmiddle" alt="Bug" style="vertical-align: text-bottom" /></a> <a href="https://dev.majordodo.org/jira/browse/MAJ-60" style="color: #3b73af; text-decoration: none">MAJ-60</a>
</td>
</tr>
<tr>
<td style="vertical-align: top;; padding: 0px; border-collapse: collapse; padding-right: 5px; font-size: 20px; line-height: 30px; mso-line-height-rule: exactly" class="page-title-pattern-header-container"> <span class="page-title-pattern-header" style="font-family: Arial, sans-serif; padding: 0; font-size: 20px; line-height: 30px; mso-text-raise: 2px; mso-line-height-rule: exactly; vertical-align: middle"> <a href="https://dev.majordodo.org/jira/browse/MAJ-60" style="color: #3b73af; text-decoration: none">FOLLOWER Broker suddently stopped and cannot recover snapshot</a> </span>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand wrapper-special-margin" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff; padding-top: 10px; padding-bottom: 5px">
<table class="keyvalue-table" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Issue Type:
</th>
<td class="has-icon" style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <img src="cid:jira-generated-image-avatar-9ea8acdd-07b2-4988-bef0-d50b79eaee64" height="16" width="16" border="0" align="absmiddle" alt="Bug" style="vertical-align: text-bottom" /> Bug
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Affects Versions:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
0.1.22
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Assignee:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <a class="user-hover" rel="eolivelli" id="email_eolivelli" href="https://dev.majordodo.org/jira/secure/ViewProfile.jspa?name=eolivelli" style="color:#3b73af;; color: #3b73af; text-decoration: none">Enrico Olivelli</a>
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Components:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
Broker
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Created:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
02/Feb/16 4:21 PM
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Environment:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top">
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">3 Broker ensemble, 1 LEADER - 2 FOLLOWER</p>
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Priority:
</th>
<td class="has-icon" style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <img src="cid:jira-generated-image-static-high-8c67899c-97ea-4514-92b3-6a565f1455e5" height="16" width="16" border="0" align="absmiddle" alt="Critical" style="vertical-align: text-bottom" /> Critical
</td>
</tr>
<tr>
<th style="color: #707070; font: normal 14px/20px Arial, sans-serif; text-align: left; vertical-align: top; padding: 2px 0">
Reporter:
</th>
<td style="padding: 0px; border-collapse: collapse; font: normal 14px/20px Arial, sans-serif; padding: 2px 0 2px 5px; vertical-align: top"> <a class="user-hover" rel="alessandro.luccaroni" id="email_alessandro.luccaroni" href="https://dev.majordodo.org/jira/secure/ViewProfile.jspa?name=alessandro.luccaroni" style="color:#3b73af;; color: #3b73af; text-decoration: none">Alessandro Luccaroni</a>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand issue-description-container" style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff; padding-top: 5px; padding-bottom: 10px">
<table class="text-paragraph-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 2px">
<tr>
<td class="text-paragraph-pattern-container mobile-resize-text " style="padding: 0px; border-collapse: collapse; padding: 0 0 10px 0">
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">Suddently both FOLLOWER broker stopped with the following stacktrace:</p>
<div class="code panel" style="border-width: 1px;; border: 1px solid #ccc; background: #f5f5f5; font-size: 12px; line-height: 1.333; font-family: monospace; border: 1px solid #ccc; -moz-border-radius: 3px 3px 3px 3px; border-radius: 3px 3px 3px 3px; margin: 9px 0">
<div class="codeContent panelContent" style="padding: 9px 12px">
<pre class="code-java" style="margin: 10px 0 0 0; max-height: 30em; overflow: auto; white-space: pre-wrap; word-wrap: normal">
INFO | jvm 1 | 2016/02/02 15:50:26 | java.lang.RuntimeException: task 1092944061 not present in brokerstatus. maybe you are recovering broken snapshot
INFO | jvm 1 | 2016/02/02 15:50:26 | at majordodo.task.BrokerStatus.applyEdit(BrokerStatus.java:598)
INFO | jvm 1 | 2016/02/02 15:50:26 | at majordodo.task.BrokerStatus.lambda$followTheLeader$13(BrokerStatus.java:344)
INFO | jvm 1 | 2016/02/02 15:50:26 | at majordodo.task.BrokerStatus$$Lambda$25/431789991.accept(Unknown Source)
INFO | jvm 1 | 2016/02/02 15:50:26 | at majordodo.replication.ReplicatedCommitLog.followTheLeader(ReplicatedCommitLog.java:867)
INFO | jvm 1 | 2016/02/02 15:50:26 | at majordodo.task.BrokerStatus.followTheLeader(BrokerStatus.java:341)
INFO | jvm 1 | 2016/02/02 15:50:26 | at majordodo.task.Broker$1.run(Broker.java:201)
INFO | jvm 1 | 2016/02/02 15:50:26 | at java.lang.<span class="code-object" style="color: #910091">Thread</span>.run(<span class="code-object" style="color: #910091">Thread</span>.java:745)
INFO | jvm 1 | 2016/02/02 15:50:27 | [MAJORDODO] broker died. requesting shutdown
INFO | jvm 1 | 2016/02/02 15:50:27 | java.lang.Exception
INFO | jvm 1 | 2016/02/02 15:50:27 | at magnews.majordodo.broker.MajordodoBrokerService.lambda$<span class="code-keyword" style="color: #000091">new</span>$1(MajordodoBrokerService.java:116)
INFO | jvm 1 | 2016/02/02 15:50:27 | at magnews.majordodo.broker.MajordodoBrokerService$$Lambda$18/230588721.run(Unknown Source)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.Broker.lambda$setBrokerDiedCallback$4(Broker.java:88)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.Broker$$Lambda$19/853072100.run(Unknown Source)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.Broker.brokerFailed(Broker.java:672)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.StatusChangesLog.signalBrokerFailed(StatusChangesLog.java:52)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.replication.ReplicatedCommitLog$2.leadershipLost(ReplicatedCommitLog.java:299)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.replication.ZKClusterManager.close(ZKClusterManager.java:308)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.replication.ReplicatedCommitLog.close(ReplicatedCommitLog.java:790)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.BrokerStatus.close(BrokerStatus.java:288)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.Broker.shutdown(Broker.java:286)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.Broker.access$1000(Broker.java:56)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.Broker$1.run(Broker.java:262)
INFO | jvm 1 | 2016/02/02 15:50:27 | at java.lang.<span class="code-object" style="color: #910091">Thread</span>.run(<span class="code-object" style="color: #910091">Thread</span>.java:745)
INFO | jvm 1 | 2016/02/02 15:50:28 | MagNewsServiceStarterWrapper stopping service (retCode=0) , pid 35421@mn1-jmsbroker2
</pre>
</div>
</div>
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">After that the FOLLOWER cannot start with a similar error:</p>
<div class="code panel" style="border-width: 1px;; border: 1px solid #ccc; background: #f5f5f5; font-size: 12px; line-height: 1.333; font-family: monospace; border: 1px solid #ccc; -moz-border-radius: 3px 3px 3px 3px; border-radius: 3px 3px 3px 3px; margin: 9px 0">
<div class="codeContent panelContent" style="padding: 9px 12px">
<pre class="code-java" style="margin: 10px 0 0 0; max-height: 30em; overflow: auto; white-space: pre-wrap; word-wrap: normal">
INFO | jvm 1 | 2016/02/02 15:50:26 | java.lang.RuntimeException: task 1092944061 not present in brokerstatus. maybe you are recovering broken snapshot
INFO | jvm 1 | 2016/02/02 15:50:26 | at majordodo.task.BrokerStatus.applyEdit(BrokerStatus.java:598)
INFO | jvm 1 | 2016/02/02 15:50:26 | at majordodo.task.BrokerStatus.lambda$followTheLeader$13(BrokerStatus.java:344)
INFO | jvm 1 | 2016/02/02 15:50:26 | at majordodo.task.BrokerStatus$$Lambda$25/431789991.accept(Unknown Source)
INFO | jvm 1 | 2016/02/02 15:50:26 | at majordodo.replication.ReplicatedCommitLog.followTheLeader(ReplicatedCommitLog.java:867)
INFO | jvm 1 | 2016/02/02 15:50:26 | at majordodo.task.BrokerStatus.followTheLeader(BrokerStatus.java:341)
INFO | jvm 1 | 2016/02/02 15:50:26 | at majordodo.task.Broker$1.run(Broker.java:201)
INFO | jvm 1 | 2016/02/02 15:50:26 | at java.lang.<span class="code-object" style="color: #910091">Thread</span>.run(<span class="code-object" style="color: #910091">Thread</span>.java:745)
INFO | jvm 1 | 2016/02/02 15:50:27 | [MAJORDODO] broker died. requesting shutdown
INFO | jvm 1 | 2016/02/02 15:50:27 | java.lang.Exception
INFO | jvm 1 | 2016/02/02 15:50:27 | at magnews.majordodo.broker.MajordodoBrokerService.lambda$<span class="code-keyword" style="color: #000091">new</span>$1(MajordodoBrokerService.java:116)
INFO | jvm 1 | 2016/02/02 15:50:27 | at magnews.majordodo.broker.MajordodoBrokerService$$Lambda$18/230588721.run(Unknown Source)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.Broker.lambda$setBrokerDiedCallback$4(Broker.java:88)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.Broker$$Lambda$19/853072100.run(Unknown Source)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.Broker.brokerFailed(Broker.java:672)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.StatusChangesLog.signalBrokerFailed(StatusChangesLog.java:52)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.replication.ReplicatedCommitLog$2.leadershipLost(ReplicatedCommitLog.java:299)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.replication.ZKClusterManager.close(ZKClusterManager.java:308)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.replication.ReplicatedCommitLog.close(ReplicatedCommitLog.java:790)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.BrokerStatus.close(BrokerStatus.java:288)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.Broker.shutdown(Broker.java:286)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.Broker.access$1000(Broker.java:56)
INFO | jvm 1 | 2016/02/02 15:50:27 | at majordodo.task.Broker$1.run(Broker.java:262)
INFO | jvm 1 | 2016/02/02 15:50:27 | at java.lang.<span class="code-object" style="color: #910091">Thread</span>.run(<span class="code-object" style="color: #910091">Thread</span>.java:745)
INFO | jvm 1 | 2016/02/02 15:50:28 | MagNewsServiceStarterWrapper stopping service (retCode=0) , pid 35421@mn1-jmsbroker2
</pre>
</div>
</div>
<p style="margin-top:0;margin-bottom:10px;; margin: 10px 0 0 0">We deleted the snapshot on one of the two FOLLOWER and it started correctly. In the mean time the other one was restarting continuisly, after 2 restart the startup completed correctly and the FOLLOWER was again attached to the LEADER. </p>
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td class="email-content-main mobile-expand " style="padding: 0px; border-collapse: collapse; border-left: 1px solid #ccc; border-right: 1px solid #ccc; border-top: 0; border-bottom: 0; padding: 0 15px 0 16px; background-color: #fff">
<table id="actions-pattern" cellspacing="0" cellpadding="0" border="0" width="100%" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 1px">
<tr>
<td id="actions-pattern-container" valign="middle" style="padding: 0px; border-collapse: collapse; padding: 10px 0 10px 24px; vertical-align: middle; padding-left: 0">
<table align="left" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td class="actions-pattern-action-icon-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 0px; vertical-align: middle"> <a href="https://dev.majordodo.org/jira/browse/MAJ-60#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none"> <img class="actions-pattern-action-icon-image" src="cid:jira-generated-image-static-comment-icon-ac853001-5fb1-4b26-83f8-8f7fb6ca03d2" alt="Add Comment" title="Add Comment" height="16" width="16" border="0" style="vertical-align: middle" /> </a>
</td>
<td class="actions-pattern-action-text-container" style="padding: 0px; border-collapse: collapse; font-family: Arial, sans-serif; font-size: 14px; line-height: 20px; mso-line-height-rule: exactly; mso-text-raise: 4px; padding-left: 5px"> <a href="https://dev.majordodo.org/jira/browse/MAJ-60#add-comment" target="_blank" title="Add Comment" style="color: #3b73af; text-decoration: none">Add Comment</a>
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
<!-- there needs to be content in the cell for it to render in some clients -->
<tr>
<td class="email-content-rounded-bottom mobile-expand" style="padding: 0px; border-collapse: collapse; color: #fff; padding: 0 15px 0 16px; height: 5px; line-height: 5px; background-color: #fff; border-top: 0; border-left: 1px solid #ccc; border-bottom: 1px solid #ccc; border-right: 1px solid #ccc; border-bottom-right-radius: 5px; border-bottom-left-radius: 5px; mso-line-height-rule: exactly">
</td>
</tr>
</table>
</td>
</tr>
<tr>
<td id="footer-pattern" style="padding: 0px; border-collapse: collapse; padding: 12px 20px">
<table id="footer-pattern-container" cellspacing="0" cellpadding="0" border="0" style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td id="footer-pattern-text" class="mobile-resize-text" width="100%" style="padding: 0px; border-collapse: collapse; color: #999; font-size: 12px; line-height: 18px; font-family: Arial, sans-serif; mso-line-height-rule: exactly; mso-text-raise: 2px">
This message was sent by Atlassian JIRA <span id="footer-build-information">(v7.0.2#70111-<span title="88534db3753fbdf1d483354f0dfe1314cfb4a534" data-commit-id="88534db3753fbdf1d483354f0dfe1314cfb4a534}">sha1:88534db</span>)</span>
</td>
<td id="footer-pattern-logo-desktop-container" valign="top" style="padding: 0px; border-collapse: collapse; padding-left: 20px; vertical-align: top">
<table style="border-collapse: collapse; mso-table-lspace: 0pt; mso-table-rspace: 0pt">
<tr>
<td id="footer-pattern-logo-desktop-padding" style="padding: 0px; border-collapse: collapse; padding-top: 3px"> <img id="footer-pattern-logo-desktop" src="cid:jira-generated-image-static-footer-desktop-logo-ba0706f5-9d72-4ee2-9719-62fc929a93bd" alt="Atlassian logo" title="Atlassian logo" width="169" height="36" class="image_fix" />
</td>
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>
</table>
</body>
</html>