Memory used does not mean memory used by your login/char/map servers.
Memory is also used for OS caching, paging, among other things, so it will increase.
The problem with most VPS is CPU contention ratio - that is, how many other VPSes are sharing the physical CPU cores.
My guess is, if it's not a memory issue (which I doubt), then it's likely a CPU resource issue.
Lastly, run some I/O tests to see what your I/O speeds are. If your system is paging from lack of physical RAM, AND your I/O is slow, then you'll experience major slowness.