Symptoms
The following errors appear in Event Viewer on the hardware node:Event ID 2020
Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2020
Description: The server was unable to allocate from the system paged pool because the pool was empty.
Event ID 2019
Event Type: Error
Event Source: Srv
Event Category: None
Event ID: 2019
Description: The server was unable to allocate from the system Non-Paged pool because the pool was empty.
Cause
The error means that the server has a shortage of non-paged (NP) memory pool. The NP pool on Windows 2003 x86 systems has a limit of 256 MB, which is used for critical kernel operations. In case the NP pool is overloaded, the system becomes slow and unresponsive and some software components cease to work normally (for example, IIS starts refusing connections).The NP memory pool shortage can be caused by memory leaks in third-party software, malware, or generally overstraining the system with resource-intensive operations.
Resolution
The problem can be tracked in the following way:1. Open Task Manager, go to the Performance tab and see the Non-paged pool usage in the Kernel Memory Usage area. Values higher than 220,000 KB are considered dangerous for the system stability (x86 platform only).
2. To find out the process that consumes the NP pool most, click the Processes tab, select View -> Select columns on the top menu, and select the PID, Session ID, and Non-paged Pool check boxes.
3. Sort the processes list by the NP Pool column and look for the process with the largest NP Pool value. Remember the PID value of this process.
4. Open the command-line prompt and use the vzquery command to find out which container runs the offending process.
vzquery p2v <PID_found_at_previous_step>5. Then you can try to determine which services belong to the process:
vzctl exec CTID tasklist /svcExample:
PID is 10872; the process is svchost.exe
vzquery p2v 10872
Process ID: 10872
Session ID: 7 (ROOT)
VE ID: 119vzctl exec 119 tasklist /svc
Image Name PID Services
========================= ========
============================================
System Idle Process 0 N/A
System 4 N/A
smss.exe 868 N/A
csrss.exe 4388 N/A
winlogon.exe 6032 N/A
services.exe 2628 Eventlog, PlugPlay
lsass.exe 3316 Netlogon, PolicyAgent, ProtectedStorage, SamSs
svchost.exe 10104 DcomLaunch
svchost.exe 8248 RpcSs
svchost.exe 3112 Dhcp, Dnscache
svchost.exe 10440 LmHosts
svchost.exe 10872 AeLookupSvc, BITS, Browser, CryptSvc,
EventSystem, helpsvc, lanmanserver,
lanmanworkstation, Messenger, Nla,
Schedule, seclogon, SENS, winmgmt, wuauserv
spoolsv.exe 11680 Spooler
msdtc.exe 12044 MSDTC
The command 'exec' is successfully finishedIt is the svchost.exe that grabs the memory from the pool. One recommendation is to disable the BITS and Automatic updates (wuauserv) services:
vzctl exec CTID sc config wuauserv start= disabled
vzctl exec CTID sc config bits start= disabled
vzctl restart CTIDNote: Another possible reason could be that the process which consumes the non-paged pool memory is a kind of malware. In this case, it is necessary to remove the malware first and then check the container with antivirus software.
Additional Information
This troubleshooting is based on the articles and tools below:Managing Container Memory Pools
Understanding Pool Consumption and Event ID: 2020 or 2019
http://msdn2.microsoft.com/en-us/library/ms792884.aspx
http://support.microsoft.com/kb/q177415/
Who's Using the Pool
The tool that can be used to evaluate memory usage is: pooltag.exe.
No
Yes