dummynet

1. Description

dummynet is a flexible tool originally designed for testing networking protocols, and since then (mis)used for bandwidth management.

It simulates/enforces queue and bandwidth limitations, delays, packet losses, and multipath effects. It also implements a variant of Weighted Fair Queueing called WF2Q+. It can be used on user's workstations, or on FreeBSD machines acting as routers or bridges.

Just to get the idea of what you can do with dummynet, e.g. by using dummynet on your workstation, or putting a PC with two ethernet cards between your network and your router and booting from the floppy-image below, here are a few examples :

These rules limit the total ICMP traffic (inbound+outbound) to 50Kbit/s

    ipfw add pipe 1 icmp from any to any
    ipfw pipe 1 config bw 50Kbit/s queue 10

These rules limit inbound traffic to 300Kbit/s for each host on your network 10.1.2.0/24.

    ipfw add pipe 2 ip from any to 10.1.2.0/24
    ipfw pipe 2 config bw 300Kbit/s queue 20 mask dst-ip 0x000000ff

If you want all machines to share evenly a single link, you should use instead:

    ipfw add queue 1 ip from any to 10.1.2.0/24
    ipfw queue 1 config weight 5 pipe 2 mask dst-ip 0x000000ff
    ipfw pipe 2 config bw 300Kbit/s

And these rules simulate an ADSL link to the moon:

    ipfw add pipe 3 ip from any to any out
    ipfw add pipe 4 ip from any to any in
    ipfw pipe 3 config bw 128Kbit/s queue 10 delay 1000ms
    ipfw pipe 4 config bw 640Kbit/s queue 30 delay 1000ms

dummynet works by intercepting packets (selected by ipfw rules - ipfw is one of the FreeBSD firewalls) in their way through the protocol stack, and passing them through one or more objects called queues and pipes, which simulate the effects of bandwidth limitations, propagation delays, bounded-size queues, packet losses, multipath. Pipes are fixed-bandwidth channels. Queues represent instead queues of packets, associated with a weight, which share the bandwidth of the pipe they are connected to proportionally to their weight.

Each pipe and queue can be configured separately, so you can apply different limitations/delays to different traffic according to the ipfw rules (e.g. selecting on protocols, addresses and ports ranges, interfaces, etc.). Pipes and queues can be created dynamically, so using a single set of rules you can apply independent limitations to all hosts in a subnet, or to all types of traffic, etc. You can also configure the system to build cascades of pipes, so you can simulate networks with multiple links and paths between source(s) and destination(s).

2. Performance, status and availability

Unlike other traffic shaping packages which run in userland, dummynet has a very little overhead, as all processing is done within the kernel. There is no data copying involved to move packets through pipes, just a bit of pointer shuffling, and the implementation is able to handle thousands of pipes with O(log N) cost, where N is the number of active pipes.

The WFQ variant we implement, called WF2Q+, has a complexity which is O(log N) in the number of active flows, so again it is able to handle efficiently thousands of flows. dummynet is part of FreeBSD since Sept.1998. It has been recently (Jan.2000 and June 2000) rewritten, so the most recent, feature-rich and robust versions are in FreeBSD 3.4-STABLE and newer releases.

You don't need to install FreeBSD on your hard disk to use it, as below you will find a bootable single-floppy version of FreeBSD which includes dummynet, bridging, and a lot of other goodies.

Dummynet is being heavily used by lots of people, and the code seems to be extremely stable and robust, especially in the 3.4-STABLE version and above. Bug fixes are generally applied to the FreeBSD source tree and are available from the CVS tree or in newer snapshot/releases of FreeBSD. From time to time i update the floppy image on this site as well.

3. Support

If you have found some bug, please report it to me by email, but don't forget to include information on which version of FreeBSD and dummynet you are using, your rules (ipfw show; ipfw pipe show), your configuration (bridge or router) etc.
If you have a simple question, again just email me and i generally try to reply as soon as possible. Again, please supply details!
For more complex things (like "i have no time to learn how to use it, i just want this work done"), or customizations and additions of new features to dummynet/ipfw, I am available (through my department) for doing support on a contract basis.
Email [email protected] for discussing details.

This said, FreeBSD users should be able to use dummynet without the need for support.
The relevant manpages (ipfw(8), dummynet(4), bridge(4)) are a great source of information, so please read updated version of them before asking questions.
You can also try posting on the various FreeBSD mailing lists or newsgroups, they are usually a very good source of information.

4. Using dummynet

Dummynet is entirely controlled by the ipfw commands and a set of sysctl variables.

4.1 Basic ipfw commands

The basic structure of ipfw commands is

   ipfw add [N] [prob X] action PROTO from SRC to DST [options]

where N is the rule number ;
X is a number between 0 and 1 that, when present, indicates the probability of getting a match on this rule if all other fields are correct. The default is deterministic match;
action is one of the actions executed on a match, which can be any of allow, deny, skipto N, pipe N and others. To send a packet to a dummynet pipe, we have to use pipe N; PROTO is the protocol type we want to match (IP, TCP, UDP, ...);
SRC and DST are address specifier (we can use addresses with netmasks and optionally followed by ports or port ranges);
options can be used to restrict the attention to packets coming from/to specific interfaces, or carrying some TCP flags or ICMP options, or bridged, etc.

4.2 Sysctl variables

The following are the main sysctl variables to control the behaviour of ipfw, bridging and dummynet:

Controlling ipfw

The firewall is mostly controlled by ipfw, and the sysctl variables only serve to give global configuration and default parameters.

net.inet.ip.fw.enable: 1
	enables firewall in the IP stack

net.inet.ip.fw.one_pass: 1
	Forces a single pass through the firewall. If set to 0,
	packets coming out of a pipe will be reinjected into the
	firewall starting with the rule after the matching one.
	NOTE: there is always one pass for bridged packets.

net.inet.ip.fw.dyn_buckets: 256 (readonly)
	Current hash table size used for dynamic rules.

net.inet.ip.fw.curr_dyn_buckets: 256
	Desired hash table size used for dynamic rules.

net.inet.ip.fw.dyn_count: 3
	Current number of dynamic rules. (readonly)
	
net.inet.ip.fw.dyn_max: 1000
	Max number of dynamic rules. If you exceed this limit, you will
	have to wait for a rule to expire before being able to create
	a new one.

net.inet.ip.fw.dyn_ack_lifetime: 300
net.inet.ip.fw.dyn_syn_lifetime: 20
net.inet.ip.fw.dyn_fin_lifetime: 20
net.inet.ip.fw.dyn_rst_lifetime: 5
net.inet.ip.fw.dyn_short_lifetime: 5
	Lifetime (in seconds) for various types of dynamic rules.

Controlling dummynet

Also dummynet is mostly controlled by ipfw, with the sysctl variables serving mostly for default parameters.

net.inet.ip.dummynet.hash_size: 64
	Size of hash table for dynamic pipes.

net.inet.ip.dummynet.expire: 1
	Delete dynamic pipes when they become empty.

net.inet.ip.dummynet.max_chain_len: 16
	Max ratio between number of dynamic queues and hash buckets.
	When you exceed (max_chain_len*buckets) queues on a pipe,
	packets not matching any of these will be all put into the
	same default queue.

Controlling bridging

Bridging is almost exclusively controlled by sysctl variables.

net.link.ether.bridge_cfg: ed2:1,rl0:1,
	set of interfaces for which bridging is enabled, and cluster
	they belong to.

net.link.ether.bridge: 0
	enable bridging.

net.link.ether.bridge_ipfw: 0
	enable ipfw for bridging.

4.3 Pipe and queue configuration

The following ipfw commands control dummynet pipes

ipfw pipe NN config ...
This command is used to create or reconfigure a pipe. NN is the numeric identifier (between 1 and 65535) of the pipe. Issuing multiple time the configuration command results in the pipe being reconfigured.
ipfw [-s field] pipe [NN] show
This command shows the parameters of a pipe. If the pipe is a dynamic one (see mask parameter), then all dynamic pipes created from this one are listed. The list can be very very long. The -s option allows you to sort the listing on one of the four counters associated to the pipe.
ipfw pipe NN delete Destroys a single pipe. Remember that packets sent to a non-existing pipe are silently dropped.
ipfw pipe flush Destroys all pipes.

The following parameters can be configured for a pipe, adding the command in the pipe config...line:

Bandwidth: bw NNunit
NN is the bandwidth assigned to the pipe, unit (which must follow the number with no intervening spaces) can be any of bit/s Kbit/s Mbit/s Byte/s KByte/s MByte/s or non-ambiguous abbreviations.
A bandwidth of 0 (or no bandwidth) results in no bandwidth limitations (hence, no queues will ever build up).
Queue size: queue NN [unit]
Sets the queue size, in slots if only NN is specified, otherwise in Bytes or KBytes. When there is no room in the queue, packets are dropped. The default queue size is 50 slots.
The combination of bandwidth and queue size influence the queueing delay. Be careful when using low bandwidths not to use too large queues, or you might end up with several seconds of queueing delay.
Also be careful when you specify the queue size in packets: if you run tests over the loopback interface, a packet can be very large, e.g. 16KB, again resulting in huge delays.
Delay: delay NN ms
Sets the propagation delay of the pipe, in milliseconds. Note that the queueing delay component is independent of the propagation delay. Also note that all delays are approximated with a granularity of 1/HZ seconds (HZ is typically 100, but we suggest using HZ=1000 and maybe even larger values).
Random Packet Loss: plr X
X is a floating point number between 0 and 1 which causes packets to be dropped at random. This is done generally to simulate lossy links. The default is 0, or no loss.
Dynamic queue creation: mask ...
It is possible to associate a mask to a pipe so that bandwidth and queue limitations are enforced separately for packets belonging to different flows.
The mask command lets you specify which parts of the following fields contribute to identify a flow:
```
[proto N] [src-ip N] [dst-ip N] [src-port N] [dst-port N]
```
where N is a bitmask where significant bits are set to 1. You can specify one or more masks, or the all keyword to mean that all fields are fully significant.
The default (when no mask are specified) is to ignore all fields, so that all packets are considered to belong to the same flow.
Whenever a new flow is encountered, a new queue (with the specified bandwidth and queue size) is created.
WARNING!!! the number of dynamic queues that can be created in this way can become very large. They are accessed through a hash table, whose size you can define using the buckets NN specifier after the mask command.

To use WF2Q+, packets must be passed to queues which in turn must be connected to a pipe.

The following ipfw commands control dummynet pipes

ipfw queue NN config ...
This command is used to create or reconfigure a queue. NN is the numeric identifier (between 1 and 65535) of the queue. Issuing multiple time the configuration command results in the queue being reconfigured.
ipfw queue NN delete Destroys a single queue. Remember that packets sent to a non-existing queue are silently dropped.
ipfw queue flush Destroys all queues.

The following parameters can be configured for a queue, adding the command in the queue config...line:

Pipe: pipe NN
NN is the identifier of the pipe used for regulating traffic.
Weight: weight NN
NN is the weight (1..100, default 1) associated to the queue.
Per-Flow queueing: mask ...
The syntax is the same as for pipes. However, all queues created dynamically will share the parent pipe's bandwidth according to the weight.
Queue size, Random Packet Loss:
Same as for pipes.

5. Using dummynet for testing protocols

Dummynet was originally created to test network protocols and applications, possibly even on a standalone system. As a consequence, some of its features such as delay emulation, random loss etc. are explicitly designed for that purpose.

There are a few things you should take in mind when doing such tests, to avoid getting incorrect results. They are all obvious things, still it is better to have them in mind.

Choosing a reasonable buffer size.
As said earlier, packet can be subject to a delay which is proportional to the total queue size (in bytes), and inversely proportional to the bandwidth. At low bandwidths, this queueing delays can be extremely high, especially if the queue size is defined in terms of packets and packets are large. The default queue size is almost certainly too large for most purposes, and it is often preferable to define the queue size in terms of bytes rather than packets.
Half-duplex vs. Full-duplex channels.
With the exception of shared-medium networks such as the ethernet, most links that you want to simulate for your experiments are full duplex. As such, the proper configuration is the following:
```
    ipfw add pipe 1 ip from A to B
    ipfw add pipe 2 ip from B to A
    ipfw pipe 1 config ...
    ipfw pipe 2 config ...
```
Should you really need to mode a half duplex network, then you can use the following sequence. But think twice before you do so, as it is often a non-realistic mode.
```
    ipfw add pipe 3 ip from A to B
    ipfw add pipe 3 ip from B to A
    ipfw pipe 3 config ...
```
Interactions between bridging and multicast
You can use ipfw (and dummynet) in a bridge by setting some sysctl variables:
```
	sysctl -w net.link.ether.bridge=1
	sysctl -w net.link.ether.bridge_ipfw=1
```
and then specify your firewall configuration.
Be careful when you run experiment involving multicast traffic through a dummynet-enabled bridge. Unless you set the rules right, multicast traffic in a bridge goes through the firewall code twice: once during forwarding at level 2, once when the packet is passed to the local IP stack of the bridge.
Starting from Feb.2000, there are to avoid this problem. One involves a sysctl variable:
```
	sysctl -w net.inet.ip.fw.enable=0
```
which avoids that the firewall is invoked at the ip level. Otherwise, you can use the bridged specifier in your ruleset to match only bridged packets:
```
    ipfw add pipe 1 ip from any to any bridged
```
Running over the loopback interface.
Dummynet was originally designed for running experiments on a standalone machine. The loopback interface lets you run senders and receivers on the same machine, but you should remember a few things:
- The firewall is invoked on all packets.
  This means that if you have a configuration such as
```
    ipfw add pipe 4 ip from 127.0.0.1 to 127.0.0.1
    ipfw pipe 4 config delay 100ms
```
  and do a simple ping 127.0.0.1 you will see a delay of approximately 400ms. In fact the ICMP request goes through the pipe twice (once down, once up), and the same for the ICMP reply. For the same reason, if you also have bandwidth or queue limitations, remember that the queue sees the traffic multiple times.
  You can partially overcome this problem by using additional ipfw options, e.g. specifying a direction for matching packets, or the uid of the sender or receiving process. Alternatively, you can assign multiple aliases to the loopback interface, and make sure that the sender and receiver bind their local endpoint to different addresses so that you will have distinct rules matching traffic in the two directions.
- The MTU of the loopback interface defaults to 16KB
  The usual default for ethernet is 1500, and for point-to-point links often smaller (576 or so). You can simply fix this by redefining the mtu to the desired value with
```
    ifconfig lo0 mtu 1500
```
TCP defaults.
Be very careful when using TCP, especially between processes running on the same machine, or on the same subnet.
Apart from the MTU issue mentioned earlier, at least on FreeBSD, TCP starts with a full window when the remote endpoint is on the same subnet as one of the local addresses. You need a simple fix in the source (tcp_input.c i believe) to fix this behaviour in FreeBSD 3.x, whereas FreeBSD 4.x has sysctl variable(s) to set the initial window.
Secondly, when you do experiments on configuration with a large delay-bandwidth product, remember that many applications use the default window size which is small, something like 16KB. You might end up not using the full bandwidth just because your data transfer is window-limited.

5.1 Simulating multipath

One nice feature of the new version of dummynet is the ability to simulate multiple paths between sender and receiver. This is done using probabilistic match, e.g.:

    ipfw add prob 0.33 pipe 1 ip from A to B
    ipfw add prob 0.5  pipe 2 ip from A to B
    ipfw add           pipe 3 ip from A to B
    ipfw pipe 1 config ...
    ipfw pipe 2 config ...
    ipfw pipe 3 config ...

Given the right packet, the first rule will match with probability 1/3; in the remaining 2/3 of occurrence we move to the second rule, which will match with prob 1/2 (so overall 1/2*1/3 = 1/3), and the remaining 1/3 of occurrence will move to the third rule, which has a deterministic match. We can then configure the three pipes as desired to emulate phenomena such as packet reordering etc.

6 Related links

Here i collect some info on how to do various ipfw-related things. Most of this is just URLs collected from the mailing list so the reliability of the info might be different (for good or bad) from what is in this page.

PPP Over Ethernet
Detailed instructions on how to set up a PPPoE connection.
ALTQ
Alternate queueing scheme.

A PicoBSD floppy

To conclude... if you want to try dummynet, here is a bootable floppy image of a system with FreeBSD, bridging, ipfw, dummynet, natd, ppp, drivers for a few interfaces, and accessible via telnet.

To setup this system, download the 1.44MB image, pico.000608.bin and copy it to a floppy using dd under FreeBSD, or rawrite under DOS/Windows.

Then put the floppy into a machine with hopefully at least one interface, and wait for it to boot. When the system comes up, login as root, password "setup", and you can play with bridging, ipfw and dummynet using the above commands.

Luigi Rizzo
Dipartimento di Ingegneria dell'Informazione -- Univ. di Pisa
via Diotisalvi 2 -- 56126 PISA
tel. +39-050-568533 Fax +39-050-568522
email: [email protected]