Network Delay Testing Using Docker and Pumba

Network Delay Testing Using Docker and Pumba

The delay of a network specifies how long it takes for a bit of data to travel across the wire from one communication endpoint to another. As your application grows network delay becomes really important design and performance characteristic, specially when your app needs to run in multiple regions of the world and synchronise data between them. For example chart of delays between AWS regions can be found here.

So how to test if your application will really work as expected? When each request sent from region one to region two will have ±200ms delay or even couple seconds of delay if network connection is poor. Let’s see how to setup local testing environment with network delays using Docker and Pumba.

Setup

First of all Docker has to be installed on your machine and your application packaged as Docker image.

codespacelab:~ lab$ docker --version
Docker version 19.03.2, build 6a30dfc

Then let’s open three terminal windows and create docker network by typing:

docker network create testnet

In one of the terminals. Validate that network is created by running:

docker network ls

For simplicity of this example instead of using some real application we will use simple Docker alpine image. Then in terminal window 1 run:

docker run -it --rm --network testnet --name test alpine sh -c "apk add --update iproute2 iperf && sh"

With this command Docker alpine image will be fetched from Docker registry. Container will join testnet network we created before, iproute2 and iperf will be updated to latest and container opened using Shell. Note: iproute2 is requited for Pumba Docker network emulator to work. Then in terminal 2 run:

docker run -it --rm --network testnet --name test1 alpine sh -c "apk add --update iproute2 iperf && sh"

Command is almost identical to the first one just it starts second container with name test1 and now we have two containers on the same network. To validate that run:

docker network inspect testnet

Output:

[
    {
        "Name": "testnet",
        "Id": "758edc8e4b369ca772e4c2b1811a2fff25eac31fb593bb0d39319365bcd5c7f7",
        "Created": "2019-10-31T11:28:43.4492961Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "07f9049f378a8a92709b44f8e95279f7da91a3cd580eadb4f25c5ff158b0a10f": {
                "Name": "test",
                "EndpointID": "fc52a2e715bef3e1c7746df7bb866aca8b1b3fee2e476d01f9ce3a59cf0d9236",
                "MacAddress": "02:42:ac:14:00:04",
                "IPv4Address": "172.20.0.4/16",
                "IPv6Address": ""
            },
            "f9d23aa4012d550c47ef37efda421b60c1fbe606fdab1c32578f8926a82c6010": {
                "Name": "test1",
                "EndpointID": "69a81c882ceafbb1e95114c3ad9ea5f18d854e7257aa099837c450c22b3c1b8b",
                "MacAddress": "02:42:ac:14:00:05",
                "IPv4Address": "172.20.0.5/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Then on container test1 we will start network listener with iperf:

iperf -s -u -i 1

On container test we will make a call to that listener:

iperf -c test1 -u

Output:

To add network delay on our setup we need to download Pumba network emulator binary from its Github repository and execute it. For example to add 5s delay for 20min on Docker container test you can run command in terminal 3:

./pumba netem --interface eth0 --duration 20m delay --time 5000 test

Call listener on test1 from test again:

iperf -c test1 -u

From output in test1 container we can see that delay was added successfully.

This simple example is only one of many possible test cases. I personally find Pumba emulator very useful while investigating data synchronisation between regions when clusters of services needs to talk to each other. In that case you can start two or more clusters on different Docker networks locally and an extra network though which they will communicate and setup delays on it.

Add Comment

Your email address will not be published. Required fields are marked *