Yury Shefer
2017-05-18 21:47:37 UTC
Hi all,
I'm trying to understand OpenVPN memory usage patterns under the load.
Is there any hidden knobs available to see where the memory gets
allocated and why it never gets released? I'm aware that the daemon is
using internal garbage collector and internal memory management
functions - that's pretty much all I know.
Here is what I noticed:
1) when I hit max-sessions and the daemon no longer accepts new
connections (by logging "new incoming connection would exceed maximum
number of clients (16200)"), the memory usage by the process
continuously increases (by ~30 MB per minute). For 16K connections
OpenVPN will consume around 3-3.2Gb of RAM and then, in a period of
~2-3 hours, it will consume the rest of VM RAM (8 GB total) and gets
killed by the kernel.
2) If I disconnect all the clients before VM runs out of RAM - the
process doesn't release the memory (which may be OK). Next, if I try
to bring up 16k clients again (without process restart) - the process
will crash due to OoM error and without reaching 16K sessions.
I tried:
1) latest OpenVPN 2.3.15 and 2.4.2 available in OpenVPN repo for
Ubuntu 16.04 - the behavior is the same.
2) run the process with valgrind (valgrind --leak-check=yes) but it
reported no errors. Logging with verb 9 with such amount of sessions
is a bit unrealistic.
3) cgroup to set process memory limit
(/sys/fs/cgroup/memory/openvpn/memory.limit_in_bytes) to something
like 4GB - it looks a bit more stable (the process runs longer) but in
the end it always gets terminated.
My test scale config is quite simple: IPv6 + TCP. No traffic over VPN
connections, I'm testing the number of sessions only.
management localhost 7505
local 172.30.30.7
port 1194
proto tcp
dev tun0
tun-ipv6
ca /etc/openvpn/crypto/ca.crt
cert /etc/openvpn/crypto/server.crt
dh /etc/openvpn/crypto/dh2048.pem
server 10.254.0 255.255.0.0
iproute /usr/local/bin/ovpn-ip
server-ipv6 dec0:ffef:1::/64
route-ipv6 dec0:ffef:1::/48
topology subnet
push route-ipv6 ::/0
client-config-dir /etc/openvpn/ccd
duplicate-cn
keepalive 240 600
reneg-sec 345600
reneg-bytes 2048000
cipher AES-256-CBC
auth SHA256
comp-lzo
max-clients 16200
user openvpn
group openvpn
persist-key
persist-tun
status /var/log/openvpn/openvpn-status.log
;crl-verify /etc/openvpn/crl/crl.pem
;log-append /var/log/openvpn/openvpn.log
suppress-timestamps
writepid /var/run/openvpn/openvpn.pid
verb 3
script-security 2
socket-flags TCP_NODELAY
push socket-flags TCP_NODELAY
Appreciate any help or suggestions!
--
Thanks,
Yury.
I'm trying to understand OpenVPN memory usage patterns under the load.
Is there any hidden knobs available to see where the memory gets
allocated and why it never gets released? I'm aware that the daemon is
using internal garbage collector and internal memory management
functions - that's pretty much all I know.
Here is what I noticed:
1) when I hit max-sessions and the daemon no longer accepts new
connections (by logging "new incoming connection would exceed maximum
number of clients (16200)"), the memory usage by the process
continuously increases (by ~30 MB per minute). For 16K connections
OpenVPN will consume around 3-3.2Gb of RAM and then, in a period of
~2-3 hours, it will consume the rest of VM RAM (8 GB total) and gets
killed by the kernel.
2) If I disconnect all the clients before VM runs out of RAM - the
process doesn't release the memory (which may be OK). Next, if I try
to bring up 16k clients again (without process restart) - the process
will crash due to OoM error and without reaching 16K sessions.
I tried:
1) latest OpenVPN 2.3.15 and 2.4.2 available in OpenVPN repo for
Ubuntu 16.04 - the behavior is the same.
2) run the process with valgrind (valgrind --leak-check=yes) but it
reported no errors. Logging with verb 9 with such amount of sessions
is a bit unrealistic.
3) cgroup to set process memory limit
(/sys/fs/cgroup/memory/openvpn/memory.limit_in_bytes) to something
like 4GB - it looks a bit more stable (the process runs longer) but in
the end it always gets terminated.
My test scale config is quite simple: IPv6 + TCP. No traffic over VPN
connections, I'm testing the number of sessions only.
management localhost 7505
local 172.30.30.7
port 1194
proto tcp
dev tun0
tun-ipv6
ca /etc/openvpn/crypto/ca.crt
cert /etc/openvpn/crypto/server.crt
dh /etc/openvpn/crypto/dh2048.pem
server 10.254.0 255.255.0.0
iproute /usr/local/bin/ovpn-ip
server-ipv6 dec0:ffef:1::/64
route-ipv6 dec0:ffef:1::/48
topology subnet
push route-ipv6 ::/0
client-config-dir /etc/openvpn/ccd
duplicate-cn
keepalive 240 600
reneg-sec 345600
reneg-bytes 2048000
cipher AES-256-CBC
auth SHA256
comp-lzo
max-clients 16200
user openvpn
group openvpn
persist-key
persist-tun
status /var/log/openvpn/openvpn-status.log
;crl-verify /etc/openvpn/crl/crl.pem
;log-append /var/log/openvpn/openvpn.log
suppress-timestamps
writepid /var/run/openvpn/openvpn.pid
verb 3
script-security 2
socket-flags TCP_NODELAY
push socket-flags TCP_NODELAY
Appreciate any help or suggestions!
--
Thanks,
Yury.