Side note: Using a secondary network interface is recommended since the following commands could make a remote machine unreachable.

This a blog post about the basics of netem and tc on how to modify the outgoing traffic. You could modify the incoming traffic with an Intermediate Functional Block pseudo-device in Linux, but I am not too familiar with it and is out of scope for now.

Reasons to simulate an unreliable network connection #

There are various reasons why you want to modify the traffic between devices. The last time we had to ensure that a streaming server in Frankfurt could handle incoming video streams with a high latency over an unreliable connection from the US. The other time we had to provide proof that some SAP modules can’t handle the additional latency of a VPN and that the problem is on their side and not ours.

Some additional reasons besides troubleshooting could be:: testing your network monitoring solution; whether your application or server handles unreliable connections well; simply do some research.

This post tries to provide enough information to help you troubleshoot various problems quickly and simulate certain scenarios.

Network shaping options

tc and netem provide are variety of options to shape the outgoing traffic.

We are going to cover the basics of the following options in this post:: adding latency; adding jitter; sending duplicated packets; adding a percentage of packet loss; sending corrupt packets

Those options can be combined and will cover most of the cases.

Basics of tc #

tc stands for ‘traffic control’ and, as the name implies, is used to configure the traffic control of the Linux kernel and is part of the iproute2 package. Netem stands for ‘network emulator’ and is controlled by the tc command.

You can check quickly whether tc is available by typing tc -V.

kuser@pleasejustwork:~$ tc -V
tc utility, iproute2-5.15.0, libbpf 0.5.0

Show or delete current options

Show the currently applied options for an interface:: tc -p qdisc ls dev eth0

kuser@pleasejustwork:~$ sudo tc -p qdisc ls dev eth0
qdisc netem 8001: root refcnt 2 limit 1000 delay 100ms  50ms

You can delete all options for a specific interface with the following command:: tc qdisc del dev eth0 root

Side note: the following options are temporary and don’t survive a reboot!

A breakdown of a common tc command can be found in the first example below.

Limiting to a specific IP or port

Unfortunately, it is not that easy to limit the applied options to a specific IP or port. It is possible, but outside the scope of this basic guide. To avoid problems, it is therefore recommended to use a secondary network interface.

I might rework this section at some point. For further reading, feel free to check the official documentation for the filters.

Units used for Parameters for the netem options #

Almost every ‘nenum’ option can have one or more parameters. I thought it would make sense to show you the available units before we start with the practical part.

Rates

The bandwidths can be specified with the following units:: bit - Bits per second; kbit - Kilobits per second; mbit - Megabits per second; gbit - Gigabits per second; tbit - Terabits per second; bps - Bytes per second; kbps - Kilobytes per second; mbps - Megabytes per second; gbps - Gigabytes per second; tbps - Terabytes per second

Time

The time for latency and other options can be specified as follows:: us - Microseconds; ms - Milliseconds; s - Seconds

Netem Options #

I am going to explain the syntax in the first scenario.

As a reference, this is the normal ICMP request/ ping over a separate interface.

kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1 
PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data.
64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=0.458 ms
64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=0.520 ms
64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=0.453 ms
64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=0.420 ms
64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=0.513 ms
64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=0.412 ms
64 bytes from 10.10.22.1: icmp_seq=7 ttl=255 time=0.550 ms
64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=0.548 ms
64 bytes from 10.10.22.1: icmp_seq=9 ttl=255 time=0.402 ms
64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=0.376 ms

--- 10.10.22.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9202ms
rtt min/avg/max/mdev = 0.376/0.465/0.550/0.060 ms

Add Latency / Delay #

The netem latency will be added to the normal latency of the connection.

DELAY := delay TIME [ JITTER [ CORRELATION ]]

sudo tc qdisc add dev eth0 root netem delay 100ms: sudo # run command as sudo; tc # command stands for ‘traffic control’; qdisc # stands for ‘Queue discipline’; add|change|del # is the action that tc should perform to a option; dev eth0 # choosing the network interface; root # qdiscs ID; netem delay 100ms # ‘netem’ option + parameter

Results

kuser@pleasejustwork:~$ sudo tc qdisc add dev eth0 root netem delay 100ms
[sudo] password for kuser: 
kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1 
PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data.
64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=101 ms
64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=7 ttl=255 time=101 ms
64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=9 ttl=255 time=100 ms
64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=100 ms

--- 10.10.22.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9013ms
rtt min/avg/max/mdev = 100.416/100.466/100.586/0.050 ms

To remove this tc rule, send the same command again, but replace add with del.

sudo tc qdisc del dev eth0 root netem delay 100ms

Add Jitter #

If you want to add more Jitter - or in other words - variance in latency, add another parameter at the end. This is a plus/minus value.

sudo tc qdisc change dev eth0 root netem delay 100ms 50ms

Results

kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1 
PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data.
64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=105 ms
64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=88.6 ms
64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=108 ms
64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=109 ms
64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=130 ms
64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=54.5 ms
64 bytes from 10.10.22.1: icmp_seq=7 ttl=255 time=141 ms
64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=102 ms
64 bytes from 10.10.22.1: icmp_seq=9 ttl=255 time=124 ms
64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=146 ms

--- 10.10.22.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9011ms
rtt min/avg/max/mdev = 54.495/110.797/145.590/25.366 ms

The added latency will be in a range from 50-150ms from now on.

Send duplicate packets #

Sending random duplicate packets over a specific interface:: sudo tc qdisc change dev eth0 root netem duplicate 1%

Simulate Packet loss #

There are various reasons for packet loss: an unreliable network connection, network congestion, bugs, and so on.

To drop random packets of a specific interface, simply use the following command:: sudo tc qdisc add dev eth0 root netem loss 20%

Results

kuser@pleasejustwork:~$ ping -c 10 -I eth0 10.10.22.1 
PING 10.10.22.1 (10.10.22.1) from 10.10.22.51 eth0: 56(84) bytes of data.
64 bytes from 10.10.22.1: icmp_seq=1 ttl=255 time=0.833 ms
64 bytes from 10.10.22.1: icmp_seq=2 ttl=255 time=0.414 ms
64 bytes from 10.10.22.1: icmp_seq=3 ttl=255 time=0.576 ms
64 bytes from 10.10.22.1: icmp_seq=4 ttl=255 time=0.443 ms
64 bytes from 10.10.22.1: icmp_seq=5 ttl=255 time=0.449 ms
64 bytes from 10.10.22.1: icmp_seq=6 ttl=255 time=0.510 ms
64 bytes from 10.10.22.1: icmp_seq=8 ttl=255 time=0.515 ms
64 bytes from 10.10.22.1: icmp_seq=10 ttl=255 time=0.302 ms

--- 10.10.22.1 ping statistics ---
10 packets transmitted, 8 received, 20% packet loss, time 9221ms
rtt min/avg/max/mdev = 0.302/0.505/0.833/0.145 ms