Rippled thread cpu usage

The cpu usage of xrp.ninja validator spikes every 2 hours. I always wonder what it is doing during the spikes. So I wrote a script to report the thead cpu usage to a statsd server. 

I started statsd/graphite/grafana using https://github.com/ripple/rippledmon

Then I wrote this script:

#!/usr/bin/python3
import collections
import itertools
import os
import re
import statsd
import subprocess
import time

HEADER = b'%CPU COMMAND'
STATSD = 'XXX'
PORT = 'YYY'
PREFIX = 'rippled.threads'
TOP = 'env HOME={0} top -b -n 1 -p $(pgrep rippled) -H'.format(os.getcwd())

def report(c, jobs):
    p = subprocess.run(TOP, shell=True, stdout=subprocess.PIPE,
            stderr=subprocess.PIPE)
    if p.returncode != 0:
        print(p.stderr)
        return
    m = collections.defaultdict(float)
    for line in itertools.dropwhile(lambda x: x != HEADER, p.stdout.splitlines()):
        if line == HEADER:
            continue
        usage, comm = line.decode().strip().split(' ', 1)
        if usage == '0.0':
            break
        comm = re.sub(r'(:|\s|#|\.)+', '_', comm)
        m[comm] += float(usage)
        if comm not in jobs:
            jobs.add(comm)
    m['sum'] = sum(m.values())
    with c.pipeline() as pipe:
        for comm in jobs:
            if comm in m:
                pipe.gauge(comm, int(m[comm] + 0.5))
            else:
                pipe.gauge(comm, 0)

def main():
    c = statsd.StatsClient(STATSD, PORT, prefix=PREFIX)
    jobs = {'sum'}
    while True:
        report(c, jobs)
        time.sleep(1)


if __name__ == '__main__':
    main()

This script uses top command to report thread cpu usage. I created a toprc which only shows cpu usage and command name (thread name if -H is used) and put the toprc in a different directory than $HOME and run the script in that directory.

The result shows SHAMapStore is the culprit.

Comments

Popular posts from this blog

The diff between the original SEC complaint against Ripple and the amended one

Send $SGB (Songbird) to another wallet using a script