XRP Ninja

As you may know, my rippled validator runs on GCE. So I use stackdriver to monitor it. I created two custom metrics for it.

One metric is uptime with rippled version as a label. Here's a live graph for this metric. From this graph, you can easily tell what the current version is and when it was upgraded to that version. You can also correlate this with other graphs to see if new version may have caused certain changes in rippled behavior.
The other metric is the percentage of current uptime that rippled has been in a certain state. Normally the curve for "full" state should asymptotically approach 100. If it is not, then rippled is in trouble.

I know I am have been too lazy. It's just a matter of copy and paste.

So thanks to any future coil subscriber who visits this blog. Your support will be greatly appreciated.

PS: here is a howto https://xrpcommunity.blog/how-to-monetize-your-content-with-coil/

https://github.com/r0bertz/ripple

I have been working on this lately. It's a fork of its original one. I learned a lot since I started working on it.

Specifically, I learned there are a lot of inconsistencies in the output of rippled API.

See for yourself in https://github.com/rubblelabs/ripple/issues/36#issuecomment-383355198

There's been some problems with the validator ever since it crashed about a week ago. It doesn't crash any more since I added much more disk space. But it easily falls behind and needs to play catch up several hours after each restart. I learned this from state_accounting field in the output of "rippled server_info". I created a metric and a chart from it. This is what the chart looks for the last week.
The "Full" mode state means it is fully synced and participating in consensus process. You can see it is now asymptotically approaching 100 now. The value means the percentage of uptime the validator stays in each mode.
I figured out why the disk was out of space. It was because the validator fell behind. When that happens, online delete is disabled. See this code:
https://github.com/ripple/rippled/blob/fc0d64f5eec4386db7146251ab1a7fe880bec17c/src/ripple/app/misc/SHAMapStoreImp.cpp#L751
I saw some "Not deleting" messages in the log which led me to t…

The above graph shows what happened.

I did get an alert from GCE that disk usage was high. I have an alert policy which says alert me if disk usage is over 80% for more than 5 minutes. However, it was too late, so I didn't get up and thought maybe it could resolve on its own. But it didn't. And GCE didn't keep alerting me, which surprises me.

Rippled logged these two lines before it died:

2018-Jan-10 11:33:27 Application:FTL Remaining free disk space is less than 512MB
2018-Jan-10 11:33:27 Application:FTL Application::onStop took 23ms
So rippled killed itself: https://github.com/ripple/rippled/search?utf8=%E2%9C%93&q=%22Remaining+free+disk+space+is+less+than+%22&type=
Before that, log was flooded with the following messages for 5 hours: 2018-Jan-10 10:55:08 LoadMonitor:WRN Job: recvGetLedger run: 1390ms wait: 0ms
2018-Jan-10 10:55:32 LoadMonitor:WRN Job: recvGetLedger run: 1250ms wait: 0ms
2018-Jan-10 10:55:32 LoadMonitor:WRN Job: recvGetLedger run: 1533ms wait: 0m…

Copied from the following link: https://www.xrpchat.com/topic/16362-rippled-0810-released/?do=findComment&comment=191441
mDuo13 wrote this. Kudos to him.
To recap the Decentralization Strategy, here's a summary: Switch to using a validator list site (vl.ripple.com). This is where we are now.All rippled instances configured to use the site can automatically follow Ripple's updates to the recommended set of validators, in lockstep.In case you're curious, the validator list site publishes cryptographically signed recommendations of validators, so it's not easy to impersonate. And rippled caches the data it gets from the site, so the XRP Ledger won't go down even if vl.ripple.com is down for a while. (It might be tough to bring new rippled servers online while vl.ripple.com is down, but I think there are some protections against that, too.)Update the site and the existing validators to use validator tokens instead of master validator secret keys.This adds security to…