Bad network performance with 10Gbit


I have trouble with the network performance inside my virtual machines.

My KVM-Host machine is connected to a 10Gbit Network. All interfaces are configured to a mtu of 4132. On this host I have no problems and I can use the full bandwidth:

iperf -c -w 65536 -p 12345 -t 60 -P4

ID] Interval Transfer Bandwidth
4] 0.0-60.0 sec 18.8 GBytes 2.69 Gbits/sec
5] 0.0-60.0 sec 15.0 GBytes 2.14 Gbits/sec
6] 0.0-60.0 sec 19.3 GBytes 2.76 Gbits/sec
3] 0.0-60.0 sec 15.1 GBytes 2.16 Gbits/sec
[SUM] 0.0-60.0 sec 68.1 GBytes 9.75 Gbits/sec

Inside a virtual machine don’t reach this result:

iperf -c -w 65536 -p 12345 -t 60 -P 4

ID] Interval Transfer Bandwidth
3] 0.0-60.0 sec 5.65 GBytes 808 Mbits/sec
4] 0.0-60.0 sec 5.52 GBytes 790 Mbits/sec
5] 0.0-60.0 sec 5.66 GBytes 811 Mbits/sec
6] 0.0-60.0 sec 5.70 GBytes 816 Mbits/sec
[SUM] 0.0-60.0 sec 22.5 GBytes 3.23 Gbits/sec

I only can use 3,23Gbits of 10Gbits. I use the virtio driver for all of my vms, but I have also tried to use the e1000 nic device instead.

With starting the iperf performance test on multiple vms simultaneously I can use the full bandwidth of the kvm host’s interface. But only one vm can’t use the full bandwith. Is this a known limitation, or can I improve this performance?

Does anyone have an idea how I can improve my network performance? It’s very important, because I want to use the network interface to boot all vms via AOE (ATA over Ethernet).

If I mount a harddisk via AOE inside a vm I get only this results:
Write |CPU |Rewrite |CPU |Read |CPU
102440 |10 |51343 |5 |104249 |3

On the KVM Host I get those results on a mouted AOE Device:
Write |CPU |Rewrite |CPU |Read |CPU
205597 |19 |139118 |11 |391316 |11

If I mount the AOE Device directly on the kvm-host and put a virtual harddisk-file in it I got the following results inside a vm using this harddisk-file:
Write |CPU |Rewrite |CPU |Read |CPU
175140 |12 |136113 |24 |599989 |29

Here more details about my environment:

KVM Version:
QEMU PC emulator version 0.12.3 (qemu-kvm-0.12.3), Copyright © 2003-2008 Fabrice Bellard

KVM Host Kernel:
2.6.32-22-server #36-Ubuntu SMP Thu Jun 3 20:38:33 UTC 2010 x86_64 GNU/Linux

KVM Host OS:
Distributor ID: Ubuntu
Description: Ubuntu 10.04 LTS
Release: 10.04
Codename: lucid

KVM Guest Kernel:
2.6.32-22-server #36-Ubuntu SMP Thu Jun 3 20:38:33 UTC 2010 x86_64 GNU/Linux

KVM Guest OS:
Distributor ID: Ubuntu
Description: Ubuntu 10.04 LTS
Release: 10.04
Codename: lucid

I would feel very happy if someone can help to solve my problem!


Hello everybody,
I now have tested a few things:

I have upgraded to a new kernel with vhost_net support (2.6.35-6) and have compiled the new qemu-kvm version from git:// … mu-kvm.git (0.12.50). But I still got the same “bad” results.

Someone told me to have a look at my cpu usage during this test. But in my opinion it seems to be ok:

top - 08:50:55 up 11:25, 1 user, load average: 0.43, 0.14, 0.05
Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.1%us, 0.4%sy, 0.0%ni, 98.5%id, 0.8%wa, 0.0%hi, 0.1%si, 0.0%st
Cpu1 : 0.2%us, 0.4%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 1023532k total, 597276k used, 426256k free, 70772k buffers
Swap: 1049592k total, 0k used, 1049592k free, 447928k cached

Tasks: 319 total, 1 running, 318 sleeping, 0 stopped, 0 zombie
Cpu0 : 9.6%us, 9.9%sy, 0.0%ni, 64.2%id, 0.0%wa, 0.0%hi, 16.2%si, 0.0%st
Cpu1 : 6.4%us, 12.4%sy, 0.0%ni, 63.0%id, 0.3%wa, 0.0%hi, 17.9%si, 0.0%st
Cpu2 : 4.6%us, 2.3%sy, 0.0%ni, 92.8%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu3 : 4.3%us, 3.0%sy, 0.0%ni, 92.1%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st
Cpu4 : 0.3%us, 3.3%sy, 0.0%ni, 95.4%id, 0.0%wa, 0.0%hi, 1.0%si, 0.0%st
Cpu5 : 1.0%us, 0.3%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.4%us, 1.6%sy, 0.0%ni, 97.5%id, 0.0%wa, 0.0%hi, 0.4%si, 0.0%st
Cpu7 : 1.0%us, 1.3%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 : 0.0%us, 1.4%sy, 0.0%ni, 98.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 0.3%us, 3.9%sy, 0.0%ni, 95.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 0.7%us, 1.9%sy, 0.0%ni, 96.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 0.7%us, 0.7%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu12 : 1.3%us, 1.3%sy, 0.0%ni, 97.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu13 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu14 : 10.2%us, 1.1%sy, 0.0%ni, 88.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu15 : 28.8%us, 5.1%sy, 0.0%ni, 66.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 33005428k total, 5292376k used, 27713052k free, 120588k buffers
Swap: 4194296k total, 0k used, 4194296k free, 828588k cached