Skip to main content

xrp.ninja Ripple validator crashed last night due to low free disk space


The above graph shows what happened.

I did get an alert from GCE that disk usage was high. I have an alert policy which says alert me if disk usage is over 80% for more than 5 minutes. However, it was too late, so I didn't get up and thought maybe it could resolve on its own. But it didn't. And GCE didn't keep alerting me, which surprises me.

Rippled logged these two lines before it died:

2018-Jan-10 11:33:27 Application:FTL Remaining free disk space is less than 512MB
2018-Jan-10 11:33:27 Application:FTL Application::onStop took 23ms

So rippled killed itself:

Before that, log was flooded with the following messages for 5 hours:
2018-Jan-10 10:55:08 LoadMonitor:WRN Job: recvGetLedger run: 1390ms wait: 0ms
2018-Jan-10 10:55:32 LoadMonitor:WRN Job: recvGetLedger run: 1250ms wait: 0ms
2018-Jan-10 10:55:32 LoadMonitor:WRN Job: recvGetLedger run: 1533ms wait: 0ms
2018-Jan-10 10:55:32 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1534ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 712ms wait: 1230ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1937ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1927ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1927ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1927ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 2ms wait: 1922ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1924ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1922ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1918ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1913ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1909ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1897ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1897ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1890ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1890ms
So it kept getting new ledgers and wrote to the db, till it filled the partition and killed itself.

I have no idea why it got so many new ledgers, as of now.

I fixed it by resizing the disk, partition and filesystem.

I have also changed the db format from RocksDB to NuDB which I should've done when I first started running it.

Comments

Popular posts from this blog

How exactly is XRP going to work as a bridging digital asset?

Maybe this is obvious all along to some people. Anyway, I just figured this out. So I thought maybe it's worth sharing.

Previously I thought the bridging consists of two transactions, both on XRP ledger. One from fiat currency A IOU to XRP, the other from XRP to fiat currency B IOU.

Then I saw this:

https://www.xrpchat.com/topic/10981-xrapid-uses-crypto-exchanges-to-route-payments-in-real-time/

So Miguel Vias said in the pilot they used liquidity on bitstamp and bitso. However it's seems the trading volume of XRP/bitso.MXN since the beginning of 2017 is so low that it's virtually no liquidity there. See this:

https://xrpcharts.ripple.com/#/markets/XRP/MXN:rG6FZ31hDHN1K5Dkbma3PSB5uVCuVVRzfn?interval=1d&range=1y&type=candlestick

Then I realized, just like bitstamp, bitso has its own exchange and the liquidity there is much better.

https://bitso.com/trade/market/xrp/mxn

So now it's clear to me there is another way for XRP to bridge cross-border payments. The originat…

Ripple released Q3 XRP market report

https://ripple.com/insights/q3-2017-xrp-markets-report/

The release caused the price to shot up vertically. Although it fell back a little shortly after.

The highlights of the report in my opinion are:
23 percent Q-o-Q volume increaseMore xRapid partnerships to be announced in Q4

You have to know these about ripple wallet and secret key

TL;DR:

It's possible to find out others' secret keys but it's so improbable that it's almost impossible. Make sure you have a good entropy source when generating your address/secret key.Use multi-sign https://ripple.com/build/how-to-multi-sign/
See the discussion here: https://www.xrpchat.com/topic/6054-questions-about-secret-keysripple-wallet-addresses/