Example configuration

EVIX supports two ways to connect:

• A local hand-off, with no tunnel at all. This is what you want if you have a VM or a port in a facility we are in: iFog (Zurich, Frankfurt), Free Range Cloud (Fremont, Vancouver), Xenyth Cloud (Toronto), or a cross-connect at Fremont or Amsterdam. Frankfurt and Vancouver are local-only and have no tunnel endpoint.
• A layer-2 tunnel to Fremont, Amsterdam, Zurich, Toronto or New Zealand. VXLAN, GRETAP, EoIP, OpenVPN and ZeroTier are supported.

Set the MTU on your EVIX interface to 1400. The fabric between our sites is itself tunnelled, so it carries about 1450 bytes; at 1500 small packets pass and large ones are silently dropped, which looks like “ping works but BGP will not establish”. Every example below assumes 1400.

Pin a fixed MAC address on your EVIX interface. VXLAN and GRETAP interfaces are assigned a new random MAC each time they are (re)created, for example after a reboot or a config reload. If your MAC changes, our route servers can keep sending to the old one until their neighbour cache re-resolves, which stalls that session (you may see a BGP Cease or the session simply drop). Setting a fixed MAC on the interface avoids this: on Linux ip link set <iface> address <aa:bb:cc:dd:ee:ff> and keep it stable.

We test on Debian and Ubuntu, VyOS and MikroTik RouterOS, but these instructions transfer to most Linux-based systems.

Connecting from an iFog VM? Ask iFog to add a dedicated EVIX interface to your VM and configure your EVIX addresses on that new interface. Do not add EVIX to your existing FogIXP interface: FogIXP is iFog’s own exchange on a different VLAN, and your frames will never reach us.

Local and VPS connections

Start here if you have a VM or a port at one of our sites. Around 40% of connections to EVIX are local rather than tunnelled, and this is the simplest way to join: there is no tunnel to build, no endpoint to agree on and nothing to keep alive. You get an interface that is already on the EVIX peering LAN, and you put your addresses on it.

This applies if you have:

• a VPS from iFog in Zurich or Frankfurt
• a VPS from Free Range Cloud in Fremont or Vancouver
• a VPS from Xenyth Cloud in Toronto
• a physical cross-connect at Fremont (Hurricane Electric FMT2) or Amsterdam (Kolo DC1)
• a VLAN or layer-2 hand-off from a carrier to our Amsterdam, Zurich or Frankfurt site

Pick Local/VPS/Other as the connection type when you apply. Frankfurt and Vancouver are local-only and offer nothing else.

Getting the interface

Ask your host for it, not us. We cannot add an interface to someone else’s VM.

iFog customers, please read this carefully, because it is the single most common reason a new peer never comes up. Ask iFog to add a dedicated EVIX interface to your VM, meaning a separate NIC on the EVIX VLAN. Then configure your EVIX addresses on that new interface. Do not add EVIX to your existing FogIXP interface: FogIXP is iFog’s own exchange on a different VLAN, and if you put our addresses there you will see no ARP or ND replies from us and the session will never establish, with nothing obviously wrong on either side.

For Free Range Cloud and Xenyth, request a complimentary cross-connect to the EVIX switch and they will attach the interface.

Configuring your addresses

We give you one IPv6 address, and an IPv4 address only if you asked for one and actually announce IPv4 space. The peering LAN is:

Note the prefix lengths: configure the whole /24 and /64, not a /31 or a /127. Every member sits on one flat layer-2 segment and reaches every other member directly, so your router needs the full prefix on-link to resolve their addresses.

There is no gateway and no default route on the peering LAN. The only things that should leave your EVIX interface are ARP, ND and BGP, plus the traffic your own routing policy sends to prefixes you learned over EVIX. Do not point a default route at the fabric and do not point static routes at other members.

Replace ens19 with whatever your host actually called the new interface, and 206.81.104.x and 2602:fed2:fff:ffff:x::xx with the addresses we sent you.

On Linux, to try it out immediately:

ip link set ens19 mtu 1400 up
ip addr add 206.81.104.x/24 dev ens19
ip addr add 2602:fed2:fff:ffff:x::xx/64 dev ens19

To make it survive a reboot on Debian or Ubuntu with netplan, in /etc/netplan/60-evix.yaml:

network:
  version: 2
  ethernets:
    ens19:
      mtu: 1400
      accept-ra: false
      addresses:
        - 206.81.104.x/24
        - 2602:fed2:fff:ffff:x::xx/64

Then netplan apply. On a system using ifupdown, in /etc/network/interfaces:

auto ens19
iface ens19 inet static
    address 206.81.104.x/24
    mtu 1400

iface ens19 inet6 static
    address 2602:fed2:fff:ffff:x::xx/64
    mtu 1400
    accept_ra 0

Set the MTU to 1400 even though there is no tunnel on your side. Your frames still cross our tunnelled fabric to reach members at other sites, and at 1500 the large ones are silently dropped: you will be able to ping the route servers but BGP will never establish.

Turn off router advertisement acceptance on this interface (accept-ra: false or accept_ra 0). It is a shared segment with a few hundred other routers on it, and you do not want to learn a default route from one of them.

On Mikrotik RouterOS, where ether2 is the EVIX interface:

/interface set ether2 mtu=1400
/ip address add address=206.81.104.x/24 interface=ether2
/ipv6 address add address=2602:fed2:fff:ffff:x::xx/64 interface=ether2 advertise=no

Checking it works

Before you configure BGP, confirm layer 2 and layer 3 are healthy:

ping -c3 206.81.104.1
ping6 -c3 2602:fed2:fff:ffff::1
ip neigh show dev ens19

That last command should list the route servers with a MAC address. If it shows FAILED or nothing at all, your frames are not reaching the fabric and no amount of BGP configuration will help. On an iFog VM that almost always means the addresses are on the wrong interface, see above. Otherwise email us and we will look from our side.

Once ping works, carry on to establishing your BGP sessions.

Configuring tunnels

Only if you do not have a local hand-off. If you have a VM or a port at one of our sites, you want local and VPS connections above instead, and you can skip this whole section.

OpenVPN + VyOS

Important: our OpenVPN server requires LZO compression (comp-lzo). Modern OpenVPN clients (2.5 and newer) do not enable it by default, so without it your tunnel connects but no traffic passes and Neighbor Solicitations go out with no replies. Enable comp-lzo on your client, or use-lzo-compression on VyOS.

NOTE: this configuration is for VyOS 1.1.8. It is the release we have actually tested, and it is old (2018); the rolling 1.2 releases changed OpenVPN bridging and this snippet will need adapting. If you get a newer VyOS working, please send us the config.

VyOS seems to have poor support for IPv6 on its OpenVPN interfaces so our solution is to create an OpenVPN TAP connection and bridge it to a standard Linux bridge which holds all the IP configuration. When looking at the config instructions you should pay attention to three addresses, your IPv4 address, your IPv6 address and your server address. The server address is the address of the closest EVIX virtual switch to yourself geographically.

You may use the following config snippet to configure your VyOS router, ensure that you have changed anything encased in <> brackets:

set interfaces bridge br0 address '<Your IPv6 Address>'
set interfaces bridge br0 address '<Your IPv4 Address>'
set interfaces bridge br0 aging '300'
set interfaces bridge br0 hello-time '2'
set interfaces bridge br0 max-age '20'
set interfaces bridge br0 priority '0'
set interfaces bridge br0 stp 'false'
set interfaces openvpn vtun0 bridge-group bridge 'br0'
set interfaces openvpn vtun0 device-type 'tap'
set interfaces openvpn vtun0 mode 'client'
set interfaces openvpn vtun0 protocol 'udp'
set interfaces openvpn vtun0 remote-host '<Server Address>'
set interfaces openvpn vtun0 tls ca-cert-file '/config/auth/openvpn/ca.crt'
set interfaces openvpn vtun0 tls cert-file '/config/auth/openvpn/<your certificate>.crt'
set interfaces openvpn vtun0 tls key-file '/config/auth/openvpn/<your key>.key'
set interfaces openvpn vtun0 'use-lzo-compression'
set protocols bgp <Your ASN> neighbor 206.81.104.1 remote-as '137933'
set protocols bgp <Your ASN> neighbor 206.81.104.1 soft-reconfiguration 'inbound'
set protocols bgp <Your ASN> neighbor 2602:fed2:fff:ffff::1 address-family ipv6-unicast soft-reconfiguration 'inbound'
set protocols bgp <Your ASN> neighbor 2602:fed2:fff:ffff::1 remote-as '137933'

Note that you will need to create the directory /config/auth/openvpn and place the ca.crt and your certificate and private key files here. These files will be provided when you join EVIX.

OpenVPN + Debian/Ubuntu

Important: our OpenVPN server requires LZO compression (comp-lzo). Modern OpenVPN clients (2.5 and newer) do not enable it by default, so without it your tunnel connects but no traffic passes and Neighbor Solicitations go out with no replies. Enable comp-lzo on your client, or use-lzo-compression on VyOS.

Connecting to EVIX via Debian should be fairly straightforward, simply install OpenVPN via apt and place the ca, certificate and key in the /etc/openvpn folder. These files will be provided when you join EVIX.

Then create a client.conf file containing the following:

client
dev tap
proto udp

remote <Server IP> 1194
resolv-retry infinite
nobind
persist-key
persist-tun

ca ca.crt
cert as_65530.crt
key as_65530.key

remote-cert-tls server
comp-lzo
verb 3

Enable the service by running systemctl enable openvpn@client and start it by running systemctl start openvpn@client. After a few seconds verify the tunnel has come up with ip addr.

Mikrotik RouterOS 6

The EoIP tunnel below works on both RouterOS 6 and 7. The BGP commands are version 6 only; on version 7 see RouterOS 7 under establishing your BGP sessions.

EVIX also supports connection via EoIP tunnels on Mikrotik RouterOS. To connect, you will need to provide your router’s public IP address. Note that your tunnel ID will be provided to you when you join EVIX. The configuration commands are as follows:

/interface eoip add !keepalive local-address=<Your public IP> name=EVIX remote-address=<Server IP> tunnel-id=<ID> 
/routing bgp instance set default as=<Your ASN> disabled=no
/routing bgp peer add instance=default name=EVIX remote-address=206.81.104.1 remote-as=137933 ttl=default
/routing bgp peer add instance=default name=EVIX remote-address=2602:fed2:fff:ffff::1 remote-as=137933 ttl=default

Please note that we do not support EoIP keepalives so be sure to disable them.

EoIP and Linux

In case you feel adventurous, you can also use an EoIP tunnel on Linux to connect to EVIX. We recommend using the inofficial EoIP kernel module by Boian Bonev: https://github.com/bbonev/eoip.

After compiling the kernel modules and the userland management utility eoip run the following commands to create the tunnel:

cd <path-to-eoip-utility>
./eoip add name eoip-evix local <your-local-ip> remote <evix-tunnel-server-ip> tunnel-id <tunnel-id>
ip link set eoip-evix up
ip addr add 206.81.104.x/24 dev eoip-evix
ip addr add 2602:fed2:fff:ffff:x::xx/64 dev eoip-evix

Your tunnel ID will be provided to you when you join EVIX.

ZeroTier

ZeroTier is a new VPN protocol designed to be zero configuration. ZeroTier is especially advantageous in that it can automatically establish tunnels to other ZeroTier users, bypassing the EVIX switch and establishing a lower-latency connection with other peers.

While the standard implementation of ZeroTier relies on ZeroTier’s root servers, EVIX has opted to create our own route servers to maintain independence. Fortunately, the configuration steps are not much different from Vanilla ZeroTier.

First install ZeroTier using the official documentation: https://www.zerotier.com/download/

Once the install has completed:

cd /var/lib/zerotier-one
mkdir moons.d
cd moons.d
wget https://evix.org/static/0000002cb385e495.moon
systemctl restart zerotier-one

Now, verify you can see the EVIX root node by running zerotier-cli listpeers, you should see something like:

200 listpeers <ztaddr> <path> <latency> <version> <role>
200 listpeers 2cb385e495 23.129.32.56/9993;4115;4115 207 1.2.12 MOON
200 listpeers 8841408a2e 46.101.160.249/9993;1538759940976;4306 17 1.1.5 PLANET
200 listpeers 9d219039f3 188.166.94.177/9993;1538759940976;4091 2 1.1.5 PLANET

The key is to see that the EVIX root server is listed as a moon. Once you have confirmed this, simply run: zerotier-cli join 2cb385e4952b3e84 to join the network. Tell us your ZeroTier ID (zerotier-cli info) and your membership will be approved by one of our staff.

Afterwards you should be able to see the zerotier network interface ztzatk5wqr and assign your EVIX IPs using the ip command:

ip addr add 206.81.104.x/24 dev ztzatk5wqr
ip addr add 2602:fed2:fff:ffff:x::xx/64 dev ztzatk5wqr

VXLAN

Virtual eXtensible Local Area Network (VXLAN) is an extension to Virtual Local Area Network (VLAN). It encapsulates a Layer 2 Ethernet frame into a UDP packet and transmits the packet over a Layer 3 network. Make sure to use a Linux kernel newer than version 3.12 since you may run into issues with IPv6 networking on older versions.

Note: The default port for VXLAN is 4789 but for technical reasons we use port 500 instead for Amsterdam and Zurich.

To create a VXLAN tunnel, run the following commands:

ip link add vxlan-evix type vxlan id <your-tunnel-id> local <your-local-ip> remote <evix-tunnel-server-ip> dstport <vxlan-port> dev eth0
ip link set vxlan-evix up
ip addr add 206.81.104.x/24 dev vxlan-evix
ip addr add 2602:fed2:fff:ffff:x::xx/64 dev vxlan-evix

Replace the VXLAN ID, remote address and interface IP with the information provided by EVIX. If needed, replace eth0 with your server’s primary interface.

GRETAP

GRETAP is GRE carrying Ethernet frames, which makes it a layer-2 tunnel where plain GRE is not. It is a good option on routers without VXLAN support.

Note: GRE is IP protocol 47 and has no port numbers, so it cannot pass through NAT and needs a clean public IPv4 endpoint. If you are behind NAT, or your endpoint is IPv6, use VXLAN instead.

The tunnel is keyless, and your tunnel ID is provided to you when you join. On Linux:

ip link add gt<your-tunnel-id> type gretap local <your-public-ipv4> remote <evix-tunnel-server-ipv4>
ip link set gt<your-tunnel-id> mtu 1400 up
ip addr add 206.81.104.x/24 dev gt<your-tunnel-id>
ip addr add 2602:fed2:fff:ffff:x::xx/64 dev gt<your-tunnel-id>

Make sure your firewall permits IP protocol 47 in both directions. If the interface comes up but you see no ARP or ND replies, that is almost always a firewall dropping protocol 47, or NAT in the path.

Establishing your BGP sessions

We run two route servers and strongly recommend connecting to both of them:

BIRD 2

This is what we run on the route servers, and the configuration we can help with best. The following example was provided by @Nicholis (many thanks!) and is a good starting point.

You probably already have a method of exporting your routes, and should easily be able to adapt the below configuration to work with that. For the purposes of this example, it is assumed that you have your prefixes defined in the static protocol my_routes, and have an export filter called export_my_routes. All you need to do is to modify the my_routes protocol to your prefixes, and replace 65001 with your ASN.

protocol static my_routes {
        ipv6;
        route 2001:db8:1::/48 reject;
        route 2001:db8:2::/48 reject;
}

filter export_my_routes {
        if proto = "my_routes" then {
                accept;
        }
        reject;
}

protocol bgp evix_01_v6 {
       local as 65001;
       neighbor 2602:fed2:fff:ffff::1 as 137933;

        ipv6 {
                 import all;
                 export filter export_my_routes;
        };
}

protocol bgp evix_02_v6 {
       local as 65001;
       neighbor 2602:fed2:fff:ffff::253 as 209762;

        ipv6 {
                 import all;
                 export filter export_my_routes;
        };
}

BIRD 1

BIRD 1 is end of life and we no longer run it anywhere. If you are still on it, we recommend moving to BIRD 2 and using the example above.

Quagga/frr

Help wanted!

Do you run Quagga or frr on EVIX? We’d appreciate if you would provide us with your (example) config so we can add it to this document: helpdesk@ev-ix.org

Mikrotik RouterOS 7

The tunnel side is unchanged: the EoIP commands above work the same on version 7. What changed is BGP. RouterOS 7 dropped /routing bgp instance and /routing bgp peer in favour of /routing/bgp/connection, so the version 6 snippet above will not apply.

Read this first, it explains most failures. Our route servers are passive: they never open a BGP connection themselves and wait for yours. So your router has to be the one that connects. In Winbox terms, on each peer leave Connect ticked; a peer that only listens will sit there forever with nothing happening on either side.

IPv4 and IPv6 are separate connections, even over a single EoIP tunnel. Dual-stack with both route servers therefore means four connections, not two. A common outcome is IPv6 coming up while IPv4 stays down because only one of them was actually configured.

The settings per connection, using the field names Winbox shows:

Remote Address is a plain host address. Note that Winbox may redisplay it with a /32 afterwards; that is cosmetic and not a problem in itself.

As a starting point on the command line:

/interface eoip set [find name=EVIX] mtu=1400
/ip address add address=206.81.104.x/24 interface=EVIX
/ipv6 address add address=2602:fed2:fff:ffff:x::xx/64 interface=EVIX advertise=no

/routing/bgp/connection add name=evix-fmt-v4 as=<your ASN> \
    local.address=206.81.104.x remote.address=206.81.104.1 remote.as=137933 \
    address-families=ip connect=yes
/routing/bgp/connection add name=evix-ams-v4 as=<your ASN> \
    local.address=206.81.104.x remote.address=206.81.104.253 remote.as=209762 \
    address-families=ip connect=yes
/routing/bgp/connection add name=evix-fmt-v6 as=<your ASN> \
    local.address=2602:fed2:fff:ffff:x::xx remote.address=2602:fed2:fff:ffff::1 \
    remote.as=137933 address-families=ipv6 connect=yes
/routing/bgp/connection add name=evix-ams-v6 as=<your ASN> \
    local.address=2602:fed2:fff:ffff:x::xx remote.address=2602:fed2:fff:ffff::253 \
    remote.as=209762 address-families=ipv6 connect=yes

Help wanted, and please read this as written.

Unlike the other examples on this page, we have not run this one ourselves: nobody on the team operates RouterOS 7. The two notes above are solid, one from our own route-server configuration and one from a real peering session, but the exact parameter names may differ on your release.

If you get RouterOS 7 working on EVIX, please send us the configuration that actually worked and we will replace this section with it, with credit: helpdesk@ev-ix.org. That is how the BIRD 2 example on this page got here.

Some helpful troubleshooting steps

In order, because these catch most cases:

1. Set your MTU to 1400. If ping works but BGP never establishes, this is almost always why. See the note at the top of this page.
2. Check that you are the side connecting. Our route servers are passive: they never open a session themselves and wait for yours. If your BGP daemon is also configured passive, or only listens, neither side will ever connect and the session sits there looking idle on both ends.
3. Check both address families. IPv4 and IPv6 are separate BGP sessions even over one tunnel. IPv6 up and IPv4 down usually means only one was configured.
4. Check our looking glass, https://lg.evix.org/. Broken sessions are listed at the bottom, often with the reason.

Then the usual checks:

• Can you ping the route servers? 2602:fed2:fff:ffff::1 and 206.81.104.1. (There is no gateway on the peering LAN: it is a flat layer-2 segment, so there is nothing else to ping through.)
• Is the neighbour resolving at layer 2? ip neigh show dev <your EVIX interface> should list the route servers with a MAC address.
• Display routes: ip route / ip -6 route
• Display BGP session information: show ip bgp summary
• Display received routes: show ip bgp neighbors <Neighbor IP> received-routes
• Display advertised routes: show ip bgp neighbors <Neighbor IP> advertised-routes

If you still have issues connecting to EVIX, please reach out to us:

• Email: helpdesk@ev-ix.org