Recently, for an Intershop on Amazon Web Services Proof of Concept, a requirement was to enable multicast traffic between two EC2 instances, each running in another availability zone. As you might know, multicast traffic is natively not supported on AWS VPC, not in an availability zone nor between availability zones.
This article explains how to enable multicast traffic between two (or more) hosts using an n2n L2 tunnel (we use CentOS 6.5).
#First; install compile tools & s3cmd yum -y install svn make gcc s3cmd #Download the code svn co https://svn.ntop.org/svn/ntop/trunk/n2n #Disable encryption and compression before compiling the binaries; this should improve performance cd n2n/n2n_v2 sed -i "s/#N2N_OPTION_AES=no/N2N_OPTION_AES=no/g" Makefile sed -i "s/#define N2N_COMPRESSION_ENABLED 1/#define N2N_COMPRESSION_ENABLED 0/g" n2n.h make
The make process should now have created two binaries:
- supernode -> to be run on the supernode, used for connection setup and VPN tunnel registration
- edge -> to be run on the nodes that will participate in multicast traffic
#Install the supernode binary on this host cp ./supernode /usr/bin/ #store binaries in s3 s3cmd put supernode s3://bucketname/files/supernode s3cmd put edge s3://bucketname/files/edge # on the edge servers, install the edge binary s3cmd get s3://bucketname/files/edge /usr/bin/edge chmod +x /usr/bin/edge
On the supernode, start the supernode process and add it to rc.local so it starts automatically when we start the instance. The deamon is listening on UDP port 1200, so don’t forget to create an AWS Security Group that allows UDP traffic on this port between all nodes that participate in the n2n tunnel (also include the supernode).
#start the supernode and automatically start it on reboot supernode -l 1200 echo "supernode -l 1200" >> /etc/rc.local
We then start the edge process on the multicast-enabled nodes. Please note that 192.168.1.1 and 192.168.1.2 are the tunnel’s endpoint addresses which will be associated with the edge0 interface created by the edge process. n2na1 is the hostname of the supernode, the -E parameter allows multicast over the tunnel:
#App server 1 edge -l n2na1:1200 -c Intershop -a 192.168.1.1 -E echo "edge -l n2na1:1200 -c Intershop -a 192.168.1.1" >> /etc/rc.local #App server 2 edge -l n2na1:1200 -c Intershop -a 192.168.1.2 -E echo "edge -l n2na1:1200 -c Intershop -a 192.168.1.2" >> /etc/rc.local
We can verify if edge was started correctly by checking if the edge0 interface was created:
root@Appa1 $ ifconfig edge0
edge0 Link encap:Ethernet HWaddr c7:5a:4b:ba:34:21
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1400 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 B) TX bytes:816 (816.0 B)
At this stage, multicast traffic between both EC2 instance over the edge0 interface should be possible. To make sure that all multicast traffic chooses the edge0 interface, we set up a static route:
#route multicast trough n2n, but during startup, wait 10 secs for the edge0 interface to become available route add -net 224.0.0.0 netmask 240.0.0.0 dev edge0 echo "sleep 10" >> /etc/rc.local echo "route add -net 224.0.0.0 netmask 240.0.0.0 dev edge0" >> /etc/rc.local
Tx to buckhill for their post on n2n
emkay
Hi there – How do we test multicasting works? I have done this with three EC2 nodes and I can’t seem to get replies when I try to ping them.
cloudar
Hi emkay,
Thanks for you reply. You can use MINT (http://sourceforge.net/projects/mc-mint/) or iperf to test if multicasting works.
iperf example:
Client; iperf -c 224.0.0.10 -b 50K -t 300 -T 5 -u 1234 -i 1 -l 136
Server : iperf -s -B 224.0.0.10 -u -i 1
Br,
Senne Vaeyens
Peter
Is it possible to run both the supernode and the edge reliably on one of the instances?
Ben Bridts
Hey Peter,
The supernode is part of the n2n tunnel by default, so it’s not needed that you run both on the same instance.
Steve
Hi,
I’d like to know how well this scales under extremely heavy loads and if the server running supernode can be load balanced under AWS?
Thanks
Ben Bridts
Hi Steve,
AWS released multicast support for Transit Gateway. If that covers your use case, I’d recommend using that: https://aws.amazon.com/about-aws/whats-new/2019/12/run-ip-multicast-workloads-aws-transit-gateway/
Since this solution uses an overlay network the scale will be limited by the performance of n2n. They try to use peer to peer connections, but your multicast traffic will eventually have to be sent over the underlying vpc network. I’d recommend doing a load test to make sure the performance matches your expectations.
If you want to load balance the supernode, you’d have to search the n2n documentation to see if that is possible.
Kind regards,
Ben