Skip to content

FIX: jirra with mysql CPU overload when leap-second happens

July 21, 2015

When leap second happens , our jirra server start overloading. All the java processes eat all the CPU time, the app is responsive, as well as the server. But still it’s overloaded.

restarting jirra -> not helping at all

restarting mysql -> not helping at all

The FIX is , the you need to manually set the date with time 00:00:00 (in our case, in yours may be not important), then manually run ntpdate pool.ntp.org and then start again the ntpd daemon. Without any services/processes restart the server is ok now, the CPU time is back as normal and the app was not disrupted in any way.

 

strace from the set date command:

strace date -s `date +”%m%d%H%M%C%y.%S”` 2>&1
execve(“/bin/date”, [“date”, “-s”, “\342\200\235072117052015.42\342\200\235”], [/* 23 vars */]) = 0
brk(0) = 0xab5000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7d4e67b000
access(“/etc/ld.so.preload”, R_OK) = -1 ENOENT (No such file or directory)
open(“/etc/ld.so.cache”, O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=20819, …}) = 0
mmap(NULL, 20819, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7d4e675000
close(3) = 0
open(“/lib64/librt.so.1”, O_RDONLY) = 3
read(3, “\177ELF\2\1\1\3>\1@!”…, 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=43832, …}) = 0
mmap(NULL, 2128816, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7d4e255000
mprotect(0x7f7d4e25c000, 2093056, PROT_NONE) = 0
mmap(0x7f7d4e45b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f7d4e45b000
close(3) = 0
open(“/lib64/libc.so.6”, O_RDONLY) = 3
read(3, “\177ELF\2\1\1\3\3>\1\360\355\1″…, 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1903208, …}) = 0
mmap(NULL, 3733672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7d4dec5000
mprotect(0x7f7d4e04b000, 2097152, PROT_NONE) = 0
mmap(0x7f7d4e24b000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x186000) = 0x7f7d4e24b000
mmap(0x7f7d4e250000, 18600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7d4e250000
close(3) = 0
open(“/lib64/libpthread.so.0”, O_RDONLY) = 3
read(3, “\177ELF\2\1\1\3>\1 \\”…, 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=138328, …}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7d4e674000
mmap(NULL, 2208672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7d4dca9000
mprotect(0x7f7d4dcc0000, 2093056, PROT_NONE) = 0
mmap(0x7f7d4debf000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f7d4debf000
mmap(0x7f7d4dec1000, 13216, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7d4dec1000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7d4dca8000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7d4dca7000
arch_prctl(ARCH_SET_FS, 0x7f7d4dca8700) = 0
mprotect(0x7f7d4debf000, 4096, PROT_READ) = 0
mprotect(0x7f7d4e24b000, 16384, PROT_READ) = 0
mprotect(0x7f7d4e45b000, 4096, PROT_READ) = 0
mprotect(0x7f7d4e67c000, 4096, PROT_READ) = 0
munmap(0x7f7d4e675000, 20819) = 0
set_tid_address(0x7f7d4dca89d0) = 26887
set_robust_list(0x7f7d4dca89e0, 0x18) = 0
futex(0x7ffff5fd1bec, FUTEX_WAKE_PRIVATE, 1) = 0

 

Info taken from:

https://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/

Advertisements

From → Crash - Quick, Linux

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: