xrp.ninja Ripple validator crashed last night due to low free disk space


The above graph shows what happened.

I did get an alert from GCE that disk usage was high. I have an alert policy which says alert me if disk usage is over 80% for more than 5 minutes. However, it was too late, so I didn't get up and thought maybe it could resolve on its own. But it didn't. And GCE didn't keep alerting me, which surprises me.

Rippled logged these two lines before it died:

2018-Jan-10 11:33:27 Application:FTL Remaining free disk space is less than 512MB
2018-Jan-10 11:33:27 Application:FTL Application::onStop took 23ms

So rippled killed itself:

Before that, log was flooded with the following messages for 5 hours:
2018-Jan-10 10:55:08 LoadMonitor:WRN Job: recvGetLedger run: 1390ms wait: 0ms
2018-Jan-10 10:55:32 LoadMonitor:WRN Job: recvGetLedger run: 1250ms wait: 0ms
2018-Jan-10 10:55:32 LoadMonitor:WRN Job: recvGetLedger run: 1533ms wait: 0ms
2018-Jan-10 10:55:32 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1534ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 712ms wait: 1230ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1937ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1927ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1927ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1927ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 2ms wait: 1922ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1924ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1922ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1918ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1913ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1909ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1897ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1897ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1890ms
2018-Jan-10 10:55:33 LoadMonitor:WRN Job: recvGetLedger run: 0ms wait: 1890ms
So it kept getting new ledgers and wrote to the db, till it filled the partition and killed itself.

I have no idea why it got so many new ledgers, as of now.

I fixed it by resizing the disk, partition and filesystem.

I have also changed the db format from RocksDB to NuDB which I should've done when I first started running it.

Comments

Post a Comment

Popular posts from this blog

The diff between the original SEC complaint against Ripple and the amended one

--- original.txt 2021-02-18 18:32:43.171832036 -0800 +++ amended.txt 2021-02-18 18:32:29.103735165 -0800 @@ -1,4 +1,4 @@ -Case 1:20-cv-10832 Document 4 Filed 12/22/20 Page 1 of 71 +Case 1:20-cv-10832-AT Document 46 Filed 02/18/21 Page 1 of 79 Richard Best Kristina Littman @@ -34,9 +34,9 @@ : ------------------------------------------------------------------------x -20 Civ. 10832 +20 Civ. 10832 (AT) ECF Case -Complaint +First Amended Complaint Jury Trial Demanded Plaintiff Securities and Exchange Commission (the “SEC”), for its Complaint against @@ -53,7 +53,7 @@ required by the federal securities laws, and no exemption from this requirement applied. 1 -Case 1:20-cv-10832 Document 4 Filed 12/22/20 Page 2 of 71 +Case 1:20-cv-10832-AT Document 46 Filed 02/18/21 Page 2 of 79 2. @@ -103,7 +103,7 @@ 2 -Case 1:20-cv-10832 Document 4 Filed 12/22/20 Page 3 of 71 +Case 1:20-cv-10832-AT Document 46 Filed 02/18/21 Page 3 of 79 VIOLATIONS 9. @@ -150,
Read more

Send $SGB (Songbird) to another wallet using a script

You need nodejs and web3 to run it:  const Web3 = require('web3'); // Variables definition const privKey = '8xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx2'; // Genesis private key const addressFrom = '0xAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'; const addressTo = '0xBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB'; const web3 = new Web3('https://songbird.towolabs.com/rpc'); // Create transaction const deploy = async () => { console.log( `Attempting to make transaction from ${addressFrom} to ${addressTo}` ); const createTransaction = await web3.eth.accounts.signTransaction( { from: addressFrom, to: addressTo, value: web3.utils.toWei('1', 'ether'), // Send 1 SGB. 'ether' is not a typo here. gasPrice: web3.utils.toWei('225', 'Gwei'), gas: '21000', }, privKey ); // Deploy transaction const createR
Read more

Rippled thread cpu usage

Image
The cpu usage of xrp.ninja validator spikes every 2 hours. I always wonder what it is doing during the spikes. So I wrote a script to report the thead cpu usage to a statsd server.  I started statsd/graphite/grafana using  https://github.com/ripple/rippledmon Then I wrote this script: #!/usr/bin/python3 import collections import itertools import os import re import statsd import subprocess import time HEADER = b'%CPU COMMAND' STATSD = 'XXX' PORT = 'YYY' PREFIX = 'rippled.threads' TOP = 'env HOME={0} top -b -n 1 -p $(pgrep rippled) -H'.format(os.getcwd()) def report(c, jobs): p = subprocess.run(TOP, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) if p.returncode != 0: print(p.stderr) return m = collections.defaultdict(float) for line in itertools.dropwhile(lambda x: x != HEADER, p.stdout.splitlines()): if line == HEADER: continue usage, comm = line.decode().str
Read more