CPU load increases permanently

[chriser]

Hi guys,
 
I am facing a really strange problem. After restarting my server, the usage of the map server is at <5% for some hours, which is normal for my playerbase.
After some days, the usage of the map server is at 50%, and after about 2 weeks at 100% permanently, including heavy lags.
Even when I start my test server and leave it running for some days, the cpu usage increases without anyone playing on the server.
 
Specs:
CPU: Intel Celeron G530 (2x 2,4 GHz) 64bit
RAM: 4GB (3,81GB total, 3,64GB used, 1,22GB cached, 64% usage)
System: Ubuntu 12.04.5 LTS

HDD: 1TB hard drive @ 7200rpm, no RAID, SMART says few bad sectors, but still green (warnings are: reallocated secor count @ 28 secotrs, current pending sector count @ 24 secotrs, both not under threshhold, so not an error, only a warning)
 
My rathena is currently on commit f7d40ba71c57945a26a621f0c6178f9e1f26197a (Date: Thu Mar 27 01:47:51 2014 -0400) with custom modifications.

 

Does anyone have the same problem or does anyone know how to resolve this issue? Are there some kind of methods to see from which part of the code the most execution time is comming?

 

EDIT: i am using gprof right now on the testserver to hopefully find the issue...

Edited October 26, 2014 by chriser

[Microsoft]

- What's the iowait?

- Have you checked if you are being attacked?

- Is the load high 24/7 or some times of the day? Do you have CPU graphs?

Are you usig OpenVZ? Because OpenVZ doesn't have very good CPU isolation and either you could face a "bad neighbor" problem, or the node is overloaded. This is something that can only be solved from your provider.

[chriser]

I have currently an iowait of 0-50%. mainly caused by jdb2/sda3-8 (according to top, htop and iotop).

I had lots of issues with this process (sometimes it has 100% iowait for minutes) but i could not figure out why. it's fine now, and i hope this will not change

iostat output:

Linux 3.2.0-60-generic 	26/10/14 	_x86_64_	(2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          35.61    0.00    3.00   14.85    0.00   46.54

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              36.34       150.25      2906.64 2696456238 52162671636

uptime:

 13:07:54 up 207 days, 17:14, 10 users,  load average: 0.69, 0.79, 0.66
 

i am not being attacked, also i would notice that currently the load is totally fine, the map-server was restarted about 10 hours ago.

i will assume that in about 4 days the load will go up to 40-50% with the same playerbase.

 

this is a dedicated machine, no virtualization.

 

(btw, harddrive is one 1TB drive, so no RAID, i edited hdd info to the first post)

Edited October 26, 2014 by chriser

[Zell]

A system or a npc maybe bad created, with loops or timers ?

[Emistry]

probably you have some scripts that consume up all your resources ??

 

try read this, maybe it might be able to help you if your situation does match any one that described by AnnieRuru in the topic.

http://www.eathena.ws/board/index.php?showtopic=181741

[chriser]

i will try and find stuff that looks like something than can build up.

my ram usage stays the same most of the time (at about 10% constantly +-1%), only cpu usage goes up.

right now, the load is at 20% again.

 

thanks for your feedback so far

 

EDIT:

 

my asrpika does the following script:

while(isequipped(2383, 2541)) { specialeffect2 546; sleep2 1000; };

but we have that since over one year and we didn't have any problems with that... could that cause the problem somehow?

 

for me it seems that some code is starting to run when a char logs in and does not stop when he logs out. it could be that one, but i could be wrong...

Edited October 29, 2014 by chriser

[chriser]

map-server load is at ~80% right now, still the same amount of players online.

 

The performance analysis with gprof did not work because the map-server stopped accepting logins after some minutes, so basically the server hung up.

 

I haven't found an issue in the scripts...

[Playtester]

Did you try to removing the "while" loop in the script (and all similar occurences). Issue still persists?

 

Loops like that could cause it.