Rippled thread cpu usage

The cpu usage of xrp.ninja validator spikes every 2 hours. I always wonder what it is doing during the spikes. So I wrote a script to report the thead cpu usage to a statsd server. 

I started statsd/graphite/grafana using https://github.com/ripple/rippledmon

Then I wrote this script:

#!/usr/bin/python3
import collections
import itertools
import os
import re
import statsd
import subprocess
import time

HEADER = b'%CPU COMMAND'
STATSD = 'XXX'
PORT = 'YYY'
PREFIX = 'rippled.threads'
TOP = 'env HOME={0} top -b -n 1 -p $(pgrep rippled) -H'.format(os.getcwd())

def report(c, jobs):
    p = subprocess.run(TOP, shell=True, stdout=subprocess.PIPE,
            stderr=subprocess.PIPE)
    if p.returncode != 0:
        print(p.stderr)
        return
    m = collections.defaultdict(float)
    for line in itertools.dropwhile(lambda x: x != HEADER, p.stdout.splitlines()):
        if line == HEADER:
            continue
        usage, comm = line.decode().strip().split(' ', 1)
        if usage == '0.0':
            break
        comm = re.sub(r'(:|\s|#|\.)+', '_', comm)
        m[comm] += float(usage)
        if comm not in jobs:
            jobs.add(comm)
    m['sum'] = sum(m.values())
    with c.pipeline() as pipe:
        for comm in jobs:
            if comm in m:
                pipe.gauge(comm, int(m[comm] + 0.5))
            else:
                pipe.gauge(comm, 0)

def main():
    c = statsd.StatsClient(STATSD, PORT, prefix=PREFIX)
    jobs = {'sum'}
    while True:
        report(c, jobs)
        time.sleep(1)


if __name__ == '__main__':
    main()

This script uses top command to report thread cpu usage. I created a toprc which only shows cpu usage and command name (thread name if -H is used) and put the toprc in a different directory than $HOME and run the script in that directory.

The result shows SHAMapStore is the culprit.