XRP Ninja

As you may know, my rippled validator runs on GCE. So I use stackdriver to monitor it. I created two custom metrics for it. One metric is uptime with rippled version as a label. Here's a live graph for this metric. From this graph, you can easily tell what the current version is and when it was upgraded to that version. You can also correlate this with other graphs to see if new version may have caused certain changes in rippled behavior. The other metric is the percentage of current uptime that rippled has been in a certain state. Normally the curve for "full" state should asymptotically approach 100. If it is not, then rippled is in trouble.

https://github.com/r0bertz/ripple I have been working on this lately. It's a fork of its original one. I learned a lot since I started working on it. Specifically, I learned there are a lot of inconsistencies in the output of rippled API. See for yourself in  https://github.com/rubblelabs/ripple/issues/36#issuecomment-383355198

There's been some problems with the validator ever since it crashed about a week ago. It doesn't crash any more since I added much more disk space. But it easily falls behind and needs to play catch up several hours after each restart. I learned this from state_accounting field in the output of "rippled server_info". I created a metric and a chart from it. This is what the chart looks for the last week. The " Full " mode state means it is fully synced and participating in consensus process. You can see it is now asymptotically approaching 100 now. The value means the percentage of uptime the validator stays in each mode. I figured out why the disk was out of space. It was because the validator fell behind. When that happens, online delete is disabled. See this code: https://github.com/ripple/rippled/blob/fc0d64f5eec4386db7146251ab1a7fe880bec17c/src/ripple/app/misc/SHAMapStoreImp.cpp#L751 I saw some "Not deleting" messages in the l

The above graph shows what happened. I did get an alert from GCE that disk usage was high. I have an alert policy which says alert me if disk usage is over 80% for more than 5 minutes. However, it was too late, so I didn't get up and thought maybe it could resolve on its own. But it didn't. And GCE didn't keep alerting me, which surprises me. Rippled logged these two lines before it died: 2018-Jan-10 11:33:27 Application:FTL Remaining free disk space is less than 512MB 2018-Jan-10 11:33:27 Application:FTL Application::onStop took 23ms So rippled killed itself: https://github.com/ripple/rippled/search?utf8=%E2%9C%93&q=%22Remaining+free+disk+space+is+less+than+%22&type= Before that, log was flooded with the following messages for 5 hours: 2018-Jan-10 10:55:08 LoadMonitor:WRN Job: recvGetLedger run: 1390ms wait: 0ms 2018-Jan-10 10:55:32 LoadMonitor:WRN Job: recvGetLedger run: 1250ms wait: 0ms 2018-Jan-10 10:55:32 LoadMonitor:WRN Job: recvGetLed

Information about this version can be found here  https://ripple.com/dev-blog/rippled-version-0-81-0/ . This happened at about 1:45pm local time. There were some behavior changes after the upgrade: