Monitoring virtual machine state with prl_perf_ctl

Resolution

There is command-line tool prl_perf_ctl which allow to see various counters for all running virtual machines and for hypervisor layer activity:

~# prl_perf_ctl --help
Performance Counters View Utility v4.0.5612.577097
Copyright 1999-2010 Parallels Holdings, Ltd. and its affiliates.

Usage: prl_perf_ctl -a | -s | -c [-l] [-n] [-d]
--help (-h) print this help message
--all (-a) dump all storages and counters
--storage (-s) filter storages by substring name
--counter (-c) filter counters by substring name
--loop (-l) output values in the loop with (sec) interval
--nozero (-n) do not output zero values (much easier to read in the loop)
--digits (-d) output digits in groups like 123,456,789 (instead of 123456789)
Example:
-c vcpu - filter all vcpu-related counters
-s XP - show only storages with 'XP' symbols in name

This command shows all non-zero counters for virtual machine with name pvcwin40w2k3:

~#prl_perf_ctl -s pvcwin40w2k3 -n Performance Counters View Utility v4.0.5612.577097
Copyright 1999-2010 Parallels Holdings, Ltd. and its affiliates.

pvcwin40w2k3 - 23:19:21
devices.ide0.aligned_writes 187345
devices.ide0.fsync 12535
devices.ide0.new_blocks_created 48
devices.ide0.read_requests 35338
devices.ide0.read_total 860475392
devices.ide0.unaligned_writes 57571
devices.ide0.write_requests 122160
devices.ide0.write_total 921349120
kernel.activity.vcpu0.guest_proc_halt 2204282
kernel.activity.vcpu0.mon_artloop_halt 12797249
kernel.activity.vcpu0.send_halt_msg 4333211
kernel.activity.vcpu0.tsc_guest 1159860294576
kernel.activity.vcpu0.tsc_host 82955064589244
kernel.activity.vcpu0.tsc_mon 488432234980
kernel.ide.irqs 293658
kernel.paging.vcpu0.cr3_change 1371186
kernel.paging.vcpu0.cr3_flushtlb 1089
kernel.paging.vcpu0.cr3_read 36109679
kernel.paging.vcpu0.cr3_write 1372275
kernel.paging.vcpu0.pf.pagefault 1395
kernel.paging.vcpu0.pf.success 1395
kernel.vcpu0.async_thread_activate 262914
kernel.vcpu0.ept.invept 2311310
kernel.vcpu0.ept.page_dirty 270437
kernel.vcpu0.guest_switch_vtx 36134124
kernel.vcpu0.int.redirect 542200
kernel.vcpu0.io_req 13135003
kernel.vcpu0.irq_process 235196
kernel.vcpu0.lapic_access 9578660
kernel.vcpu0.mode_switch 1094
kernel.vcpu0.mon_ret_to_host 5714468
kernel.vcpu0.npt.fault 1260156
kernel.vcpu0.npt.map 796209
kernel.vcpu0.pic.setintrequest 2994638
kernel.vcpu0.sendapirequest 576139
kernel.vcpu0.tools_api_request 300346
kernel.ws.pagein 1348351
kernel.ws.pageout 1208693
kernel.ws.pages 139658
net.nic0.activate 29736
net.nic0.bcast_in 67377
net.nic0.bytes_in 9510113
net.nic0.bytes_out 2219895
net.nic0.mcast_in 34442
net.nic0.pkt_err_security 1
net.nic0.pkts_in 125041
net.nic0.pkts_out 29969
video.frames 10061

It is possible to check virtual machine state dynamically by monitoring counters changes in a loop. For example, virtual network adapter activity can be checked using this command:

~# prl_perf_ctl -s pvcwin40w2k3 -n -l -c nic
Performance Counters View Utility v4.0.5612.577097
Copyright 1999-2010 Parallels Holdings, Ltd. and its affiliates.

pvcwin40w2k3 - 23:25:58 +1 sec
net.nic0.bcast_in 10
net.nic0.bytes_in 1050
net.nic0.bytes_out 62
net.nic0.mcast_in 4
net.nic0.pkts_in 15
net.nic0.pkts_out 1
pvcwin40w2k3 - 23:25:59 +1 sec
net.nic0.activate 1
net.nic0.bcast_in 12
net.nic0.bytes_in 908
net.nic0.mcast_in 1
net.nic0.pkts_in 13
pvcwin40w2k3 - 23:26:00 +1 sec
net.nic0.bcast_in 8
net.nic0.bytes_in 522
net.nic0.bytes_out 60
net.nic0.pkts_in 9
net.nic0.pkts_out 1
pvcwin40w2k3 - 23:26:01 +1 sec
net.nic0.bcast_in 2
net.nic0.bytes_in 180
net.nic0.mcast_in 1
net.nic0.pkts_in 3

Finally, it is possible to check dispatcher activity itself:

~#prl_perf_ctl -c mgmt -l
Performance Counters View Utility v4.0.5612.577097
Copyright 1999-2010 Parallels Holdings, Ltd. and its affiliates.

disp_server - 23:30:00 +1 sec
mgmt.commands 0
mgmt.error_commands 0
disp_server - 23:30:01 +1 sec
mgmt.commands 3
mgmt.error_commands 0
disp_server - 23:30:02 +1 sec
mgmt.commands 11
mgmt.error_commands 0
disp_server - 23:30:03 +1 sec
mgmt.commands 7
mgmt.error_commands 0

Non-zero values show the amount of calls to corresponding subsystem made within time of one loop interval (in the above example - 1 second).