[Openvpn-users] Multicast fragmenting not handled by OpenVPN ?

[James MacLean]

Hi Folks,

I am seeing a problem which occurs both OpenVPN, GRE and mrouted's
builtin tunnels.

If a large Multicast packet gets sent to the tunnel, it gets fragmented,
or atleast broken up :). But when it comes out the other side, it appears
to never get reconstructed into it original format.

For some reason, this makes tunneled connections get a limited max
bandwidth of around 270Kbs. Almost like the kernel is getting slowed down
by them?

If I set the MTU at the originating machine down to something smaller
like 1100, Linux fragements the packets at the source, they pass through
the tunnel un-split, and appear to work fine end to end.

This is not only OpenVPN but also on GRE tunnels and mrouted builtin
tunnels. GRE and mrouted both use ipip.o in the Linux kernel if that
matters.

VIC and Rat appear to not be affected by this because their packets are
mostly smaller around 512 bytes I think.

To see it in action, Get mp4live running (part of mpeg4ip.sourceforge.net)
in multicast through a tunnel. I tried both mrouted and pimd. The results
will be that the fastest throughput you'll get is around 270Kbs :).

Maybe this is just particular to Linux? Or it is expected?

OpenVPN 1.4.1 and RedHat 9.0.

JES

[James Yonan]

James,

OpenVPN normally leaves all fragmenting and routing issues up to the kernel.

However OpenVPN 1.4.1 has a new experimental mode that does fragmenting
itself, allowing the MTU of the TUN/TAP device to be much larger than the MTU
of the UDP connection that carries the tunnel data.

To enable this feature, you must build OpenVPN with ./configure
--enable-mtu-dynamic

Then you can use the --mtu-disc (Linux only) or --mtu-dynamic options to
explicitly control path MTU discovery and fragmenting options.

For example,

openvpn --tun-mtu 1500 --mtu-disc yes --mtu-dynamic 500 500 [other options]

would create a tunnel that looks like it has an MTU of 1500 to the OS, but
OpenVPN would actually break the packets up so that the encrypted UDP
datagrams sent between the OpenVPN peers would never be larger than 500 bytes
(not including the IP header). The "--mtu-disc yes" option tells the OS to
set the "Don't Fragment" bit on the UDP datagrams. The downside of
--mtu-dynamic is that it will always be measureably less efficient than not
fragmenting in the first place.

The --mtu-dynamic option is still experimental at this point and has two
planned features which have not been implemented yet: (a) give --mtu-dynamic a
lower and upper bound on UDP MTU size and have OpenVPN automatically choose
the largest size (using a cryptographically secure handshake) which will not
fragment, without depending on the OSes implementation of Path MTU discovery,
and (b) given our empirically and securely determined path MTU, generate our
own "Fragmentation needed but DF set" ICMP messages to bounce back over the
TUN device, effectively constraining upstream senders to an MTU which will not
cause fragmentation.

Anyway, I mention this because it might be an interesting experiment to
isolate where the problem is occurring, i.e. if --mtu-dynamic fixes the
problem, then it may point to some kind issue with multicast fragmentation in
the kernel.

I also checked the Linux kernel source, and if you look at icmp_send in icmp.c
you will see:

/*
* No replies to physical multicast/broadcast
*/
if (skb_in->pkt_type!=PACKET_HOST)
return;

/*
* Now check at the protocol level
*/
if (rt->rt_flags&(RTCF_BROADCAST|RTCF_MULTICAST))
return;

It seems that this might create complications for Path MTU discovery on
multicast streams, because no "fragmentation needed but DF set" ICMP message
could be returned in response to a multicast stream that demands fragmentation.

James

Post by James MacLean
Hi Folks,
I am seeing a problem which occurs both OpenVPN, GRE and mrouted's
builtin tunnels.
If a large Multicast packet gets sent to the tunnel, it gets fragmented,
or atleast broken up :). But when it comes out the other side, it appears
to never get reconstructed into it original format.
For some reason, this makes tunneled connections get a limited max
bandwidth of around 270Kbs. Almost like the kernel is getting slowed down
by them?
If I set the MTU at the originating machine down to something smaller
like 1100, Linux fragements the packets at the source, they pass through
the tunnel un-split, and appear to work fine end to end.
This is not only OpenVPN but also on GRE tunnels and mrouted builtin
tunnels. GRE and mrouted both use ipip.o in the Linux kernel if that
matters.
VIC and Rat appear to not be affected by this because their packets are
mostly smaller around 512 bytes I think.
To see it in action, Get mp4live running (part of mpeg4ip.sourceforge.net)
in multicast through a tunnel. I tried both mrouted and pimd. The results
will be that the fastest throughput you'll get is around 270Kbs :).
Maybe this is just particular to Linux? Or it is expected?
OpenVPN 1.4.1 and RedHat 9.0.
JES
--
Department of Education
Nova Scotia, Canada
B3M 4B2
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
Openvpn-users mailing list
https://lists.sourceforge.net/lists/listinfo/openvpn-users

--

[James MacLean]

Hi James,

I want to say thanks to the responses I have received and the information
they have included to help me understand what is going on. I have some
feedback which will show my lack of understanding, but might help better
define the problem :).

I was unable to get the tunnel to forward large packets when using the
suggests parms (--tun-mtu 1500 --mtu-disc yes --mtu-dynamic 500 500). If I
removed the --mtu-disc yes, I was atleast seeing some packets traverse the
tunnel, but no amount to say it was working :(. I also tried setting
--tun-mtu 1500 down to 1300, and --mtu-dynamic 500 500 to 1500 with no
luck. I do believe I had properly compiled in --enable-mtu-dynamic, but if
there is a test to prove that, let me know.

Now, with the tunnel set up as it previously was, without the extra
parameters, I am seeing in the tunnel :

13:58:53.165345 198.166.79.70.35480 > 238.230.120.206.28376: udp 849 (DF)
13:58:53.187482 198.166.79.70.35480 > 238.230.120.206.28378: udp 1045 (DF)

Showing the pieces of the full packets traversing the tunnel with the DF
flag set. No Fragmentation bits set as there is when you actually fragment
the packets at the source. ie :

13:20:29.339380 198.166.79.70.35480 > 238.230.120.206.28378: udp 1022 (DF)
13:20:29.350812 198.166.79.70 > 238.230.120.206: (frag 38028:***@1080)
13:20:29.350886 198.166.79.70.35480 > 238.230.120.206.28376: udp 1472 (frag 38028:***@0+)

Now, I expect MTU discovery is useless in a multicast situation
as multicast is all one way and it has no way of knowing exactly
end-to-end what MTU's it might pass through? Example being my case where
the tunneling is there, but the originating box has no idea that it is
there or what it is doing?

Also, if the kernel is fraging the packets to fit them through the tunnel,
shouldn't they look like the frag packets when they go the physical
interface? The example above is typical of OpenVPN, GRE, and the mrouted
tunneling as far as what is sniffed off the tunnel.

Also, Dick St.Peters <***@NetHeaven.com> pointed out that if the
tunnel is just breaking apart bigger packets into smaller ones to fit
through the tunnel (not offically fragging them), it should reassemble
them on the other side ( I hope I paraphrased Dick ok). These packets
though appear back on the ethernet in their "shredded" size, not
reassembled. Is this correct?

Hope I have the right info for these questions,
JES

Post by James Yonan
James,
OpenVPN normally leaves all fragmenting and routing issues up to the kernel.
However OpenVPN 1.4.1 has a new experimental mode that does fragmenting
itself, allowing the MTU of the TUN/TAP device to be much larger than the MTU
of the UDP connection that carries the tunnel data.
To enable this feature, you must build OpenVPN with ./configure
--enable-mtu-dynamic
Then you can use the --mtu-disc (Linux only) or --mtu-dynamic options to
explicitly control path MTU discovery and fragmenting options.
For example,
openvpn --tun-mtu 1500 --mtu-disc yes --mtu-dynamic 500 500 [other options]
would create a tunnel that looks like it has an MTU of 1500 to the OS, but
OpenVPN would actually break the packets up so that the encrypted UDP
datagrams sent between the OpenVPN peers would never be larger than 500 bytes
(not including the IP header). The "--mtu-disc yes" option tells the OS to
set the "Don't Fragment" bit on the UDP datagrams. The downside of
--mtu-dynamic is that it will always be measureably less efficient than not
fragmenting in the first place.
The --mtu-dynamic option is still experimental at this point and has two
planned features which have not been implemented yet: (a) give --mtu-dynamic a
lower and upper bound on UDP MTU size and have OpenVPN automatically choose
the largest size (using a cryptographically secure handshake) which will not
fragment, without depending on the OSes implementation of Path MTU discovery,
and (b) given our empirically and securely determined path MTU, generate our
own "Fragmentation needed but DF set" ICMP messages to bounce back over the
TUN device, effectively constraining upstream senders to an MTU which will not
cause fragmentation.
Anyway, I mention this because it might be an interesting experiment to
isolate where the problem is occurring, i.e. if --mtu-dynamic fixes the
problem, then it may point to some kind issue with multicast fragmentation in
the kernel.
I also checked the Linux kernel source, and if you look at icmp_send in icmp.c
/*
* No replies to physical multicast/broadcast
*/
if (skb_in->pkt_type!=PACKET_HOST)
return;
/*
* Now check at the protocol level
*/
if (rt->rt_flags&(RTCF_BROADCAST|RTCF_MULTICAST))
return;
It seems that this might create complications for Path MTU discovery on
multicast streams, because no "fragmentation needed but DF set" ICMP message
could be returned in response to a multicast stream that demands fragmentation.
James

Post by James MacLean
Hi Folks,
I am seeing a problem which occurs both OpenVPN, GRE and mrouted's
builtin tunnels.
If a large Multicast packet gets sent to the tunnel, it gets fragmented,
or atleast broken up :). But when it comes out the other side, it appears
to never get reconstructed into it original format.
For some reason, this makes tunneled connections get a limited max
bandwidth of around 270Kbs. Almost like the kernel is getting slowed down
by them?
If I set the MTU at the originating machine down to something smaller
like 1100, Linux fragements the packets at the source, they pass through
the tunnel un-split, and appear to work fine end to end.
This is not only OpenVPN but also on GRE tunnels and mrouted builtin
tunnels. GRE and mrouted both use ipip.o in the Linux kernel if that
matters.
VIC and Rat appear to not be affected by this because their packets are
mostly smaller around 512 bytes I think.
To see it in action, Get mp4live running (part of mpeg4ip.sourceforge.net)
in multicast through a tunnel. I tried both mrouted and pimd. The results
will be that the fastest throughput you'll get is around 270Kbs :).
Maybe this is just particular to Linux? Or it is expected?
OpenVPN 1.4.1 and RedHat 9.0.
JES
--
Department of Education
Nova Scotia, Canada
B3M 4B2
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
Openvpn-users mailing list
https://lists.sourceforge.net/lists/listinfo/openvpn-users

[James MacLean]

Hi Folks,

Wanted to report that openvpn-1.4.1.3 does properly get the multicast
packets through using the suggestion from below. It's the only tunnel that
does it.

I did not get any response to my last E-mail with regards to what OpenVPN
and/or the kernel is supose to be doing with the normal settings when a
large packet has to be squeezed through the tunnel. If it should get
fragmented, it does not appear to be setting up the packets as fragmented
packets. If it is supose to break them up to get through the tunnel, that
also does not appear to be happening, as whatever is sliced up in the
tunnel ends up on the outgoing ethernet also sliced up.

I am hoping someone can explain this as it appears to be common to OpenVPN
in it's normal mode, GRE tunnels, and mrouted's tunnel. Also, the overall
effect here is to slow traffic leaving the outgoing ethernet down to
around 270Kbs, as if it is causing the kernel to get stalled.

Happy camper now to see 1 tunnel solution that will work with large
packets effectively ;).

JES

Post by James Yonan
James,
OpenVPN normally leaves all fragmenting and routing issues up to the kernel.
However OpenVPN 1.4.1 has a new experimental mode that does fragmenting
itself, allowing the MTU of the TUN/TAP device to be much larger than the MTU
of the UDP connection that carries the tunnel data.
To enable this feature, you must build OpenVPN with ./configure
--enable-mtu-dynamic
Then you can use the --mtu-disc (Linux only) or --mtu-dynamic options to
explicitly control path MTU discovery and fragmenting options.
For example,
openvpn --tun-mtu 1500 --mtu-disc yes --mtu-dynamic 500 500 [other options]
would create a tunnel that looks like it has an MTU of 1500 to the OS, but
OpenVPN would actually break the packets up so that the encrypted UDP
datagrams sent between the OpenVPN peers would never be larger than 500 bytes
(not including the IP header). The "--mtu-disc yes" option tells the OS to
set the "Don't Fragment" bit on the UDP datagrams. The downside of
--mtu-dynamic is that it will always be measureably less efficient than not
fragmenting in the first place.
The --mtu-dynamic option is still experimental at this point and has two
planned features which have not been implemented yet: (a) give --mtu-dynamic a
lower and upper bound on UDP MTU size and have OpenVPN automatically choose
the largest size (using a cryptographically secure handshake) which will not
fragment, without depending on the OSes implementation of Path MTU discovery,
and (b) given our empirically and securely determined path MTU, generate our
own "Fragmentation needed but DF set" ICMP messages to bounce back over the
TUN device, effectively constraining upstream senders to an MTU which will not
cause fragmentation.
Anyway, I mention this because it might be an interesting experiment to
isolate where the problem is occurring, i.e. if --mtu-dynamic fixes the
problem, then it may point to some kind issue with multicast fragmentation in
the kernel.
I also checked the Linux kernel source, and if you look at icmp_send in icmp.c
/*
* No replies to physical multicast/broadcast
*/
if (skb_in->pkt_type!=PACKET_HOST)
return;
/*
* Now check at the protocol level
*/
if (rt->rt_flags&(RTCF_BROADCAST|RTCF_MULTICAST))
return;
It seems that this might create complications for Path MTU discovery on
multicast streams, because no "fragmentation needed but DF set" ICMP message
could be returned in response to a multicast stream that demands fragmentation.
James

Post by James MacLean
Hi Folks,
I am seeing a problem which occurs both OpenVPN, GRE and mrouted's
builtin tunnels.
If a large Multicast packet gets sent to the tunnel, it gets fragmented,
or atleast broken up :). But when it comes out the other side, it appears
to never get reconstructed into it original format.
For some reason, this makes tunneled connections get a limited max
bandwidth of around 270Kbs. Almost like the kernel is getting slowed down
by them?
If I set the MTU at the originating machine down to something smaller
like 1100, Linux fragements the packets at the source, they pass through
the tunnel un-split, and appear to work fine end to end.
This is not only OpenVPN but also on GRE tunnels and mrouted builtin
tunnels. GRE and mrouted both use ipip.o in the Linux kernel if that
matters.
VIC and Rat appear to not be affected by this because their packets are
mostly smaller around 512 bytes I think.
To see it in action, Get mp4live running (part of mpeg4ip.sourceforge.net)
in multicast through a tunnel. I tried both mrouted and pimd. The results
will be that the fastest throughput you'll get is around 270Kbs :).
Maybe this is just particular to Linux? Or it is expected?
OpenVPN 1.4.1 and RedHat 9.0.
JES
--
Department of Education
Nova Scotia, Canada
B3M 4B2
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
Openvpn-users mailing list
https://lists.sourceforge.net/lists/listinfo/openvpn-users

[James Yonan]

Post by James MacLean
Hi Folks,
Wanted to report that openvpn-1.4.1.3 does properly get the multicast
packets through using the suggestion from below. It's the only tunnel that
does it.

Great, I'm glad it's working for you.

I should note that those of you who want to try OpenVPN's internal fragmenting
feature (which is still somewhat experimental at this point) should use
openvpn-1.4.1.3 or later and not openvpn-1.4.1. You must run ./configure with
the --enable-mtu-dynamic flag, then use the --mtu-dynamic option in OpenVPN.

The typical usage of --mtu-dynamic would be a scenario where you want to
bridge an ethernet segment using a TAP virtual device, but where IP
fragmentation is broken on the tunnel path. Because ethernet uses a fixed MTU
of 1500, when you encapsulate a 1500 byte ethernet frame into an encrypted UDP
tunnel packet, the added overhead of encryption and encapsulation will push
the resulting packet size to >1500 which will therefore require fragmentation.
If IP fragmentation is working between the hosts, the following would be
sufficient:

openvpn --dev tap --up ./mktap --remote [other-openvpn-host] --secret key
--tun-mtu 1500 --tun-mtu-extra 64 --comp-lzo

where mktap is a script:

# local = local IP endpoint for tunnel
ifconfig $1 $local netmask 255.255.255.0 mtu $2

Then you would need to bridge $1 (which will be something like tap0) with your
ethernet LAN, using a tool such as linux's brctl.

On the other hand, if IP fragmentation is broken between the hosts, adding
"--mtu-dynamic 1450 1450" will cause OpenVPN to internally fragment packets so
that the tunnel UDP datagrams never exceed 1450 bytes (not including the IP
header). This fixes the problem because UDP datagrams <=1450 will usually not
trigger IP fragmentation.

Post by James MacLean
I did not get any response to my last E-mail with regards to what OpenVPN
and/or the kernel is supose to be doing with the normal settings when a
large packet has to be squeezed through the tunnel. If it should get
fragmented, it does not appear to be setting up the packets as fragmented
packets. If it is supose to break them up to get through the tunnel, that
also does not appear to be happening, as whatever is sliced up in the
tunnel ends up on the outgoing ethernet also sliced up.

When you say "normal settings" I assume you mean without --mtu-dynamic. In a
typical fragmentation scenario as handled by the kernel, OpenVPN (or any other
app that sends large UDP packets) will try to send a UDP packet to the remote
host which exceeds the Path MTU. The kernel figures out the path MTU by using
"Path MTU Discovery" (see appropriate RFC), and will then fragment the packet
so that it doesn't exceed the Path MTU size. Path MTU discovery requires that
routers along the path properly interpret the "Don't Fragment" flag in the IP
header and that they return "Fragmentation needed but DF set" ICMP messages to
the sender. Normally it is better to leave fragmentation up to the kernel,
because fragmenting in user-space is less efficient. However there are enough
broken routers and incorrectly configured firewalls out there that Path MTU
discovery and IP fragmentation doesn't work as reliably as it should.
--mtu-dynamic is designed as a workaround for such cases.

Now in a multicast setting, Path MTU discovery isn't going to work because
multicast is by nature a one-way communication flow. As you can see in the
linux kernel snippet below, linux will refuse to send an ICMP message in
response to a multicast packet. However, if you are tunneling a multicast
stream through an OpenVPN tunnel, that stream will be tunnelled as a series of
UDP packets which can then benefit from Path MTU discovery and IP
fragmentation as it is implemented for the UDP layer.

Post by James MacLean
I am hoping someone can explain this as it appears to be common to OpenVPN
in it's normal mode, GRE tunnels, and mrouted's tunnel. Also, the overall
effect here is to slow traffic leaving the outgoing ethernet down to
around 270Kbs, as if it is causing the kernel to get stalled.

Bottom line is that most tunnels will break if you try to bridge ethernet
segments over a tunnel where Path MTU Discovery and IP Fragmentation are not
working correctly. tcpdump is a good tool to ascertain whether this is the case.

Post by James MacLean
Happy camper now to see 1 tunnel solution that will work with large
packets effectively ;).

Excellent!

James

Post by James MacLean

Post by James Yonan
James,
OpenVPN normally leaves all fragmenting and routing issues up to the kernel.
However OpenVPN 1.4.1 has a new experimental mode that does fragmenting
itself, allowing the MTU of the TUN/TAP device to be much larger than the MTU
of the UDP connection that carries the tunnel data.
To enable this feature, you must build OpenVPN with ./configure
--enable-mtu-dynamic
Then you can use the --mtu-disc (Linux only) or --mtu-dynamic options to
explicitly control path MTU discovery and fragmenting options.
For example,
openvpn --tun-mtu 1500 --mtu-disc yes --mtu-dynamic 500 500 [other options]
would create a tunnel that looks like it has an MTU of 1500 to the OS, but
OpenVPN would actually break the packets up so that the encrypted UDP
datagrams sent between the OpenVPN peers would never be larger than 500 bytes
(not including the IP header). The "--mtu-disc yes" option tells the OS to
set the "Don't Fragment" bit on the UDP datagrams. The downside of
--mtu-dynamic is that it will always be measureably less efficient than not
fragmenting in the first place.
The --mtu-dynamic option is still experimental at this point and has two
planned features which have not been implemented yet: (a) give --mtu-dynamic a
lower and upper bound on UDP MTU size and have OpenVPN automatically choose
the largest size (using a cryptographically secure handshake) which will not
fragment, without depending on the OSes implementation of Path MTU discovery,
and (b) given our empirically and securely determined path MTU, generate our
own "Fragmentation needed but DF set" ICMP messages to bounce back over the
TUN device, effectively constraining upstream senders to an MTU which will not
cause fragmentation.
Anyway, I mention this because it might be an interesting experiment to
isolate where the problem is occurring, i.e. if --mtu-dynamic fixes the
problem, then it may point to some kind issue with multicast fragmentation in
the kernel.
I also checked the Linux kernel source, and if you look at icmp_send in icmp.c
/*
* No replies to physical multicast/broadcast
*/
if (skb_in->pkt_type!=PACKET_HOST)
return;
/*
* Now check at the protocol level
*/
if (rt->rt_flags&(RTCF_BROADCAST|RTCF_MULTICAST))
return;
It seems that this might create complications for Path MTU discovery on
multicast streams, because no "fragmentation needed but DF set" ICMP message
could be returned in response to a multicast stream that demands

fragmentation.

Post by James MacLean

Post by James Yonan
James

Post by James MacLean
Hi Folks,
I am seeing a problem which occurs both OpenVPN, GRE and mrouted's
builtin tunnels.
If a large Multicast packet gets sent to the tunnel, it gets fragmented,
or atleast broken up :). But when it comes out the other side, it appears
to never get reconstructed into it original format.
For some reason, this makes tunneled connections get a limited max
bandwidth of around 270Kbs. Almost like the kernel is getting slowed down
by them?
If I set the MTU at the originating machine down to something smaller
like 1100, Linux fragements the packets at the source, they pass through
the tunnel un-split, and appear to work fine end to end.
This is not only OpenVPN but also on GRE tunnels and mrouted builtin
tunnels. GRE and mrouted both use ipip.o in the Linux kernel if that
matters.
VIC and Rat appear to not be affected by this because their packets are
mostly smaller around 512 bytes I think.
To see it in action, Get mp4live running (part of mpeg4ip.sourceforge.net)
in multicast through a tunnel. I tried both mrouted and pimd. The results
will be that the fastest throughput you'll get is around 270Kbs :).
Maybe this is just particular to Linux? Or it is expected?
OpenVPN 1.4.1 and RedHat 9.0.
JES
--
Department of Education
Nova Scotia, Canada
B3M 4B2
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
Openvpn-users mailing list
https://lists.sourceforge.net/lists/listinfo/openvpn-users

--
Department of Education
Nova Scotia, Canada
B3M 4B2
-------------------------------------------------------
This SF.net email is sponsored by: eBay
Get office equipment for less on eBay!
http://adfarm.mediaplex.com/ad/ck/711-11697-6916-5
_______________________________________________
Openvpn-users mailing list
https://lists.sourceforge.net/lists/listinfo/openvpn-users

--

[James MacLean]

Post by James Yonan

Post by James MacLean
Hi Folks,
Wanted to report that openvpn-1.4.1.3 does properly get the multicast
packets through using the suggestion from below. It's the only tunnel that
does it.

Great, I'm glad it's working for you.

Well... :) It was/is working... At work, between 2 machines both with
linux 2.4.20 type kernels and using the openvpn tun interfaces for mrouted
to route over (no mrouted own tunnel involved).

But from work to home, which is from a 2.2.x kernel to a 2.4.20 ish
kernel, it does not appear to be doing its magic. Maybe I should not have
enabled pthread in openvpn... Also, from home to work it does mrouted
tunneling over the openvpn tunnel. So it is a little different, but all I
see coming through tun0 and dropping onto eth1 is a pile of DF packets,
and all not near the 1470 that get created at the other end. So, something
is a miss along the way.

Post by James Yonan
The typical usage of --mtu-dynamic would be a scenario where you want to
bridge an ethernet segment using a TAP virtual device, but where IP
fragmentation is broken on the tunnel path. Because ethernet uses a fixed MTU
of 1500, when you encapsulate a 1500 byte ethernet frame into an encrypted UDP
tunnel packet, the added overhead of encryption and encapsulation will push
the resulting packet size to >1500 which will therefore require fragmentation.
If IP fragmentation is working between the hosts, the following would be
openvpn --dev tap --up ./mktap --remote [other-openvpn-host] --secret key
--tun-mtu 1500 --tun-mtu-extra 64 --comp-lzo
# local = local IP endpoint for tunnel
ifconfig $1 $local netmask 255.255.255.0 mtu $2
Then you would need to bridge $1 (which will be something like tap0) with your
ethernet LAN, using a tool such as linux's brctl.

And this certainly doesn't work. Not one packet makes it off eth1 after
coming through the tunnel that was originally large (ie 1470 bytes). But
again, this is along the lines of what I am commenting on right from the
start. The packets all hit the first end of the tunnel broken up with the
DF set (and in this non-working case I'm gonna be bold a blame it on the
kernel as it is mrouted's tunneling that is sending it's payload over the
tunnel) wosh through the tunnel out the other end and onto eth1, DF still
set and the packets crippled. Bandwith suffering with a max around 270Kbs.
It's so bad the streaming app (mp4live) doesn't perform.

Post by James Yonan
On the other hand, if IP fragmentation is broken between the hosts, adding
"--mtu-dynamic 1450 1450" will cause OpenVPN to internally fragment packets so
that the tunnel UDP datagrams never exceed 1450 bytes (not including the IP
header). This fixes the problem because UDP datagrams <=1450 will usually not
trigger IP fragmentation.

This one works at work. Doesn't work to home, but I am strongly starting
to believe its mrouted's tunneling, which is using the kernel that is
causing my grief...

Post by James Yonan

Post by James MacLean
I did not get any response to my last E-mail with regards to what OpenVPN
and/or the kernel is supose to be doing with the normal settings when a
large packet has to be squeezed through the tunnel. If it should get
fragmented, it does not appear to be setting up the packets as fragmented
packets. If it is supose to break them up to get through the tunnel, that
also does not appear to be happening, as whatever is sliced up in the
tunnel ends up on the outgoing ethernet also sliced up.

When you say "normal settings" I assume you mean without --mtu-dynamic.

Yes.

Post by James Yonan
Now in a multicast setting, Path MTU discovery isn't going to work because
multicast is by nature a one-way communication flow. As you can see in the
linux kernel snippet below, linux will refuse to send an ICMP message in
response to a multicast packet. However, if you are tunneling a multicast
stream through an OpenVPN tunnel, that stream will be tunnelled as a series of
UDP packets which can then benefit from Path MTU discovery and IP
fragmentation as it is implemented for the UDP layer.

Sure, the UDP packets of the tunnel can negotiate MTU... But still the
underlying MultiCast is not being helped out.

Post by James Yonan

Post by James MacLean
I am hoping someone can explain this as it appears to be common to OpenVPN
in it's normal mode, GRE tunnels, and mrouted's tunnel. Also, the overall
effect here is to slow traffic leaving the outgoing ethernet down to
around 270Kbs, as if it is causing the kernel to get stalled.

Bottom line is that most tunnels will break if you try to bridge ethernet
segments over a tunnel where Path MTU Discovery and IP Fragmentation are not
working correctly. tcpdump is a good tool to ascertain whether this is the case.

Then as a bottom line :), if a packet arrives at a tunnel and that packet
is bigger than the tunnel can deal with, then the kernel can not properly
break it up into fragments and send them through the tunnel. Openvpn can
:).

If it is just a packet traveling between normal interfaces (eth0, etc...)
it will be properly fragmented and passed through by the kernel.

So bottom line is if you have any applications that send packets through
the tunnel where the packets are bigger than what the tunnel can handle as
a single packet, they will be somehow crippled and sent through the tunnel
with the DF set and cause the tunnel to max sending this large payload at
about 270 Kbs. Atleast for MultiCasts packets.

Can't say I would expect Linux to behave this way.

Post by James Yonan

Post by James MacLean
Happy camper now to see 1 tunnel solution that will work with large
packets effectively ;).

Excellent!

Except now I have to find a way to remove any kernel tunneling apps that
might be in use :(.

JES

[James Yonan]

Post by James MacLean

Post by James Yonan

Post by James MacLean
Hi Folks,
Wanted to report that openvpn-1.4.1.3 does properly get the multicast
packets through using the suggestion from below. It's the only tunnel that
does it.

Great, I'm glad it's working for you.

Well... :) It was/is working... At work, between 2 machines both with
linux 2.4.20 type kernels and using the openvpn tun interfaces for mrouted
to route over (no mrouted own tunnel involved).
But from work to home, which is from a 2.2.x kernel to a 2.4.20 ish
kernel, it does not appear to be doing its magic. Maybe I should not have
enabled pthread in openvpn...

I doubt that pthread is an issue here. It doesn't really do anything unless
you are negotiating the tunnel with TLS, though it wouldn't hurt to rule it
out by building without pthread.

Post by James MacLean
Also, from home to work it does mrouted
tunneling over the openvpn tunnel. So it is a little different, but all I
see coming through tun0 and dropping onto eth1 is a pile of DF packets,
and all not near the 1470 that get created at the other end. So, something
is a miss along the way.

I've never used mrouted, though I understand that it uses IP over IP tunnels,
and so is likely using IP fragmentation as well. If you look at the source
snippet below, you will see that if the linux kernel encounters a multicast
packet larger than the MTU that the destination can support, it drops it into
a black hole. The IP over IP code appears to take the path MTU of the
destination referenced by the top-level encapsulation and subtract the IP over
IP header size to derive the MTU of the inner encapsulation. I would guess
that someone trying to send a packet to the endpoint of an IP over IP tunnel
will cause the kernel to see this lesser MTU and fragment according to the
fragmentability of the particular IP protocol that's being tunneled. In the
cast of multicast, it could end up routing a large packet to /dev/null, since
multicast doesn't fragment.

OpenVPN's internal fragmentation, on the other hand, blindly fragments every
packet larger than the max-size specified in the --mtu-dynamic option.

Post by James MacLean

Post by James Yonan
The typical usage of --mtu-dynamic would be a scenario where you want to
bridge an ethernet segment using a TAP virtual device, but where IP
fragmentation is broken on the tunnel path. Because ethernet uses a fixed MTU
of 1500, when you encapsulate a 1500 byte ethernet frame into an encrypted UDP
tunnel packet, the added overhead of encryption and encapsulation will push
the resulting packet size to >1500 which will therefore require fragmentation.
If IP fragmentation is working between the hosts, the following would be
openvpn --dev tap --up ./mktap --remote [other-openvpn-host] --secret key
--tun-mtu 1500 --tun-mtu-extra 64 --comp-lzo
# local = local IP endpoint for tunnel
ifconfig $1 $local netmask 255.255.255.0 mtu $2
Then you would need to bridge $1 (which will be something like tap0) with your
ethernet LAN, using a tool such as linux's brctl.

And this certainly doesn't work. Not one packet makes it off eth1 after
coming through the tunnel that was originally large (ie 1470 bytes). But
again, this is along the lines of what I am commenting on right from the
start. The packets all hit the first end of the tunnel broken up with the
DF set (and in this non-working case I'm gonna be bold a blame it on the
kernel as it is mrouted's tunneling that is sending it's payload over the
tunnel) wosh through the tunnel out the other end and onto eth1, DF still
set and the packets crippled. Bandwith suffering with a max around 270Kbs.
It's so bad the streaming app (mp4live) doesn't perform.

Can you configure mrouted to use openvpn tunnels directly, rather than routing
an IP over IP tunnel through the openvpn tunnel?

I suspect (as I believe you do) that the IP over IP tunnel might be killing
the large multicast packets before they ever reach the local endpoint of the
openvpn tunnel.

Post by James MacLean

Post by James Yonan
On the other hand, if IP fragmentation is broken between the hosts, adding
"--mtu-dynamic 1450 1450" will cause OpenVPN to internally fragment packets so
that the tunnel UDP datagrams never exceed 1450 bytes (not including the IP
header). This fixes the problem because UDP datagrams <=1450 will usually not
trigger IP fragmentation.

This one works at work. Doesn't work to home, but I am strongly starting
to believe its mrouted's tunneling, which is using the kernel that is
causing my grief...

Post by James Yonan

Post by James MacLean
I did not get any response to my last E-mail with regards to what OpenVPN
and/or the kernel is supose to be doing with the normal settings when a
large packet has to be squeezed through the tunnel. If it should get
fragmented, it does not appear to be setting up the packets as fragmented
packets. If it is supose to break them up to get through the tunnel, that
also does not appear to be happening, as whatever is sliced up in the
tunnel ends up on the outgoing ethernet also sliced up.

When you say "normal settings" I assume you mean without --mtu-dynamic.

Yes.

Post by James Yonan
Now in a multicast setting, Path MTU discovery isn't going to work because
multicast is by nature a one-way communication flow. As you can see in the
linux kernel snippet below, linux will refuse to send an ICMP message in
response to a multicast packet. However, if you are tunneling a multicast
stream through an OpenVPN tunnel, that stream will be tunnelled as a series of
UDP packets which can then benefit from Path MTU discovery and IP
fragmentation as it is implemented for the UDP layer.

Sure, the UDP packets of the tunnel can negotiate MTU... But still the
underlying MultiCast is not being helped out.

How large are the multicast packets? Do you have any control over their size?

Post by James MacLean

Post by James Yonan

Post by James MacLean
I am hoping someone can explain this as it appears to be common to OpenVPN
in it's normal mode, GRE tunnels, and mrouted's tunnel. Also, the overall
effect here is to slow traffic leaving the outgoing ethernet down to
around 270Kbs, as if it is causing the kernel to get stalled.

Bottom line is that most tunnels will break if you try to bridge ethernet
segments over a tunnel where Path MTU Discovery and IP Fragmentation are not
working correctly. tcpdump is a good tool to ascertain whether this is

the case.

Post by James MacLean
Then as a bottom line :), if a packet arrives at a tunnel and that packet
is bigger than the tunnel can deal with, then the kernel can not properly
break it up into fragments and send them through the tunnel. Openvpn can
:).
If it is just a packet traveling between normal interfaces (eth0, etc...)
it will be properly fragmented and passed through by the kernel.
So bottom line is if you have any applications that send packets through
the tunnel where the packets are bigger than what the tunnel can handle as
a single packet, they will be somehow crippled and sent through the tunnel
with the DF set and cause the tunnel to max sending this large payload at
about 270 Kbs. Atleast for MultiCasts packets.
Can't say I would expect Linux to behave this way.

Well here's the linux source (2.4.19). You not only cannot fragment
multicasts, but you cannot report your inability to do so via the ICMP mechanism.

if (skb->len+encap > rt->u.dst.pmtu && (ntohs(iph->frag_off) & IP_DF)) {
/* Do not fragment multicasts. Alas, IPv4 does not
allow to send ICMP, so that packets will disappear
to blackhole.
*/

IP_INC_STATS_BH(IpFragFails);
ip_rt_put(rt);
return;
}

Post by James MacLean

Post by James Yonan

Post by James MacLean
Happy camper now to see 1 tunnel solution that will work with large
packets effectively ;).

Excellent!

Except now I have to find a way to remove any kernel tunneling apps that
might be in use :(.
JES
--
Department of Education
Nova Scotia, Canada
B3M 4B2

--

[James MacLean]

Post by James Yonan
I've never used mrouted, though I understand that it uses IP over IP tunnels,
and so is likely using IP fragmentation as well. If you look at the source
snippet below, you will see that if the linux kernel encounters a multicast
packet larger than the MTU that the destination can support, it drops it into
a black hole. The IP over IP code appears to take the path MTU of the
destination referenced by the top-level encapsulation and subtract the IP over
IP header size to derive the MTU of the inner encapsulation. I would guess
that someone trying to send a packet to the endpoint of an IP over IP tunnel
will cause the kernel to see this lesser MTU and fragment according to the
fragmentability of the particular IP protocol that's being tunneled. In the
cast of multicast, it could end up routing a large packet to /dev/null, since
multicast doesn't fragment.

Excellent. _That_ explains it all. It explains why all kernel tunnels
suffer from this. Yes, mrouted uses IP over IP, GRE tunnels should be
similar and they would all be falling into the hole you found in the
kernel.

I expect unless you have some end-to-end smarts on the tunnel, you can not
break up packets, send them down the tunnel and reconstruct them at the
other end. That is magic only being seen in OpenVPN :).

I expect there must be some RFC that states that these MultiCast packets
should be droped and not fragmented at tunnels? I ask because if you send
a 1500 byte MultiCast packet from an application on a box that has an MTU
of 1400, it _does_ fragment it out the interface and appears to allow the
application to function. So it would appear to me that the tunnels
_should_ allow the same capabilities? Or could at least have been setup to
act as other interfaces?

Post by James Yonan
Can you configure mrouted to use openvpn tunnels directly, rather than routing
an IP over IP tunnel through the openvpn tunnel?

That's what the setup at work between the two macines is. I will be trying
the same for my home connection, but it will take some time to change the
setup,

Post by James Yonan
I suspect (as I believe you do) that the IP over IP tunnel might be killing
the large multicast packets before they ever reach the local endpoint of the
openvpn tunnel.

Yes. It was the explaination I was looking for all along :-). Many thanks
for opening my eyes :).

Post by James Yonan

Post by James MacLean
Sure, the UDP packets of the tunnel can negotiate MTU... But still the
underlying MultiCast is not being helped out.

How large are the multicast packets? Do you have any control over their size?

The packets are usually maxing around 1470. I can set them smaller using
mp4live's rtpPayloadSize setting, but I was looking to understand the
problem as it exists and see if it should be fixed at the source first.
Actually the smaller the size mp4live sends, the poorer the picture
looks... But that might be me just starting to see things :(.

Post by James Yonan
Well here's the linux source (2.4.19). You not only cannot fragment
multicasts, but you cannot report your inability to do so via the ICMP mechanism.
if (skb->len+encap > rt->u.dst.pmtu && (ntohs(iph->frag_off) & IP_DF)) {
/* Do not fragment multicasts. Alas, IPv4 does not
allow to send ICMP, so that packets will disappear
to blackhole.
*/
IP_INC_STATS_BH(IpFragFails);
ip_rt_put(rt);
return;
}

Is this code saying that if the DF bit is already set, you can not
fragment it again, or that if the DF is set, it wants to request to be
fragmented, but this code disallows it?

Sorry the questions are simple, but I'm still trying to understand why
multicast packets _can_ be fragmented when they are push from an
application out its eth0, but not when they are shoved down a tunnel.

I guess I am just missing the connecting piece. Probably that the above
code deals with packets traversing a system (in one interface out the
other) and my example of the application getting fragmented is because it
is just going out an interface, so the logic is in a different part of the
network code?

thanks again for explaining what is going on,
JES

[James MacLean]

Hi Folks,

Post by James Yonan

Post by James MacLean

Post by James Yonan

Post by James MacLean
Hi Folks,
Wanted to report that openvpn-1.4.1.3 does properly get the multicast
packets through using the suggestion from below. It's the only tunnel that
does it.

Great, I'm glad it's working for you.

Well... :) It was/is working... At work, between 2 machines both with
linux 2.4.20 type kernels and using the openvpn tun interfaces for mrouted
to route over (no mrouted own tunnel involved).
But from work to home, which is from a 2.2.x kernel to a 2.4.20 ish
kernel, it does not appear to be doing its magic. Maybe I should not have
enabled pthread in openvpn...

I doubt that pthread is an issue here. It doesn't really do anything unless
you are negotiating the tunnel with TLS, though it wouldn't hurt to rule it
out by building without pthread.

Have now tested that when I remove the mrouted tunnel and just use OpenVPN
with either mtu-dynamic or tun-extra-mtu then the multicast packets
successfully make it to their destination.

Pavlin Radoslavov (maintaner of PIMd) realized though that the Linux
kernel is setting the DF bit on _all_ multicast packets it creates. He
does not believe this to be correct. Not sure if it is backed by any RFCs
or other decisions?

Last, openvpn-1.4.1.3 runs slower then its predecessor. Or at least it is
missing something. Tried pthreads, no pthreads, with dynamic-mtu, without.
In all cases, openvpn-1.4.0 carried our traffic to its destinations
faster. There are different kinds of traffic being sent, but NFS is one
obvious example where speed was noticeably slower. Also pings used during
Nagios monitoring would be lost quite often, even when the link was not
busy. Not sure what this deals with but wanted to pass it along.

JES

[James Yonan]

Post by James MacLean
Hi Folks,

Post by James Yonan

Post by James MacLean

Post by James Yonan

Post by James MacLean
Hi Folks,
Wanted to report that openvpn-1.4.1.3 does properly get the multicast
packets through using the suggestion from below. It's the only

tunnel that

Post by James MacLean

Post by James Yonan

Post by James MacLean

Post by James Yonan

Post by James MacLean
does it.

Great, I'm glad it's working for you.

Well... :) It was/is working... At work, between 2 machines both with
linux 2.4.20 type kernels and using the openvpn tun interfaces for mrouted
to route over (no mrouted own tunnel involved).
But from work to home, which is from a 2.2.x kernel to a 2.4.20 ish
kernel, it does not appear to be doing its magic. Maybe I should not have
enabled pthread in openvpn...

I doubt that pthread is an issue here. It doesn't really do anything unless
you are negotiating the tunnel with TLS, though it wouldn't hurt to rule it
out by building without pthread.

Have now tested that when I remove the mrouted tunnel and just use OpenVPN
with either mtu-dynamic or tun-extra-mtu then the multicast packets
successfully make it to their destination.
Pavlin Radoslavov (maintaner of PIMd) realized though that the Linux
kernel is setting the DF bit on _all_ multicast packets it creates. He
does not believe this to be correct. Not sure if it is backed by any RFCs
or other decisions?
Last, openvpn-1.4.1.3 runs slower then its predecessor. Or at least it is
missing something. Tried pthreads, no pthreads, with dynamic-mtu, without.
In all cases, openvpn-1.4.0 carried our traffic to its destinations
faster. There are different kinds of traffic being sent, but NFS is one
obvious example where speed was noticeably slower. Also pings used during
Nagios monitoring would be lost quite often, even when the link was not
busy. Not sure what this deals with but wanted to pass it along.

I would like to isolate where the slowdown is occurring.

If you build 1.4.1.3 without --enable-mtu-dynamic, how does it compare with
1.4.1 also built without --enable-mtu-dynamic?

You mention 1.4.0 in the above paragraph. Does 1.4.0 perform differently than
1.4.1 (without --enable-mtu-dynamic on both)?

Now if you compare 1.4.1.3 built with --enable-mtu-dynamic, but without using
the --mtu-dynamic option, what is the result of a comparison with 1.4.1 or
1.4.0 built without --enable-mtu-dynamic?

All of the above tests should produce results that are roughly equal. If they
don't, it suggests a possible problem.

The only measurable difference in speed should occur when you are comparing an
--mtu-dynamic tunnel against a tunnel without --mtu-dynamic and where
fragmentation is not occurring.

Let me know,

James