Packet Mangling for Fun & Profit

volleyballbeginnerΔίκτυα και Επικοινωνίες

27 Οκτ 2013 (πριν από 4 χρόνια και 8 μήνες)

131 εμφανίσεις

Packet Mangling for Fun & Profit

A Brief Intro to Netfilter, User
Mode Linux, and the Linux TCP/IP


SplitTCP Summary (Context)

Netfilter Introduction

One buffer to rule them all: sk_buffs

User Mode Linux

Current Status

SplitTCP In a Nutshell

Work started by Michalis, Srikanth, and

The idea is to add transport layer proxies to
long (in terms of hop count) TCP connections.

If a link fails (due to mobility, etc.) the packet
can be re
transmitted by the proxy closest to
the failure.

A “demo
quality” implementation was done
which made many simplifying assumptions.

So What?

My current task is to take that demo
implementation and turn it into a
general implementation.

This presentation discusses the lessons
learned up to this point in that process.

Design Goals

Any and all modifications made to the
behavior of TCP should be backwards

I.e. nodes should interoperate regardless
of weather or not they’re “split tcp

If possible, no changes should be made
to the kernel proper. It’s just better for
everyone that way…

Enter Netfilter

Fortunately, Linux has a plugin API,
called Netfilter, which allows kernel
modules to hook into strategic spots in
the networking stack.

Unfortunately, in the grand tradition of
open source, documentation takes a
back seat to implementation

there’s a non
trivial learning curve.

Sideline: Plugin APIs

An example of the Delegation design
pattern (GoF)

a work unit is passed
through a series of cooperating steps to
produce the final result.

A well
designed plugin architecture can
facilitate otherwise impossible tasks and
simply possible ones.

Sideline: Kernel Modules

Most of the linux kernel can be built as
a module

a bit of code that’s loaded
into (and unloaded from) the kernel

Device drivers, file systems, even
networking protocols (IPX, Appletalk)
are candidates for modularization.

Writing a Module

Just define a few preprocessor macros

Include a few header files
(linux/module.h, linux/version.h,

And honor a small interface
(module_init(), module_exit())

You’ll have a no
op module.

Netfilter Overview

Initial design by Paul “Rusty” Russell

Netfilter is a series of callback functions
within the network stack. The API is
portable and appeared in linux

Each protocol has it’s own set of
callback points. We care about IPv4.

Netfilter Concepts

A module expresses interest in being
invoked at an arbitrary subset of the
available callback points

the function and the (global) priority in
which it should be called.

That function is passed (among other
things) a pointer to a pointer to a
packet buffer ( sk_buff ** ).

Return Values

The netfilter function has five possible
return values:

NF_ACCEPT: continue callback chain

NF_DROP: drop the packet and stop the

NF_STOLEN: stop the chain

NF_QUEUE: send the packet to userspace

NF_REPEAT: call the hook again

Netfilter Hooks in IPv

Routing Engine

Local Sockets








Say that Again?


any received packet which checksums OK.


packets destined for local sockets


foreign packets being forwarded


any outbound packet


packets originating from local sockets

Routing Engine

Local Sockets







An sk_what??

Linux uses a structure called an sk_buff
to store packet data internally.

It contains a handful of pointers to
other structures as well as a packet
data region.


Sk_buff’s and you

The data area is like a stack, only you
can insert at the head and the tail

The kernel provides a handful of helper
functions to manage sk_buff’s and their
data areas.

The various header pointers point into
the data area

which can be thought of
as a serialized packet.

Why do We Care?

An sk_buff is built as a packet travels
down the stack

each layer (TCP, IP,
Ethernet) adds their own special sauce.

This means that each header is
“squashed” in against the next

while modifying existing data is
relatively easy, adding new header data
is a bit trickier.

Don’t leave me in suspense…

Basically, you make a copy of the
sk_buff, and ask it to “grow” a bit
during the copy.

Once you have the copy

you “slide”
the IP and TCP headers backwards a
bit, insert the new option bytes, and re
checksum the packet.

A tale of “n” checksum’s

Sounds easy, right? Remember that
these sk_buffs are built one layer at a

There is no nice friendly function which
will take a TCP sk_buff and compute all
the needed checksums.

Funny thing about checksums

isn’t good enough.

Ok, insmod and <BOOM>

Remember developing on a system
without memory protection and having
to reboot ?

Kernel modules execute in kernel space

so no one’s watching your back. If
you goof, it’s time to reboot.

User Mode Linux to the rescue

User Mode Linux (UML)

A kernel patch that allows running the
linux kernel as a user
mode process on
a linux machine.

If you crash the user
mode kernel, you
just restart the process, no reboot

Where do I sign up?

Setup is (in principal) fairly easy

only it
turns out that the standard distribution
doesn’t have netfilter enabled.

So I re
built with the appropriate options and
placed the binaries in ~swift/user

There’s a README there

some support
executables must be installed as root on your

That’s It?

Not quite

you also need a file system to
boot this kernel off.

Good News: you can download a file system

you don’t have to make one.

Bad News: it’s really big. (up to 700MB)

Good News: you can share one among many

Bad News: you have to read the HOWTO.

Building Modules Under UML

Building a kernel module is the same
under UML as under “KML”

it is
important that you build it against the
source used to build the target kernel.

For “our” UML build

that source is in

Current Status

A skeleton of a splittcp module (tcpproxy.c)

It can inspect locally generated packets and
add our newly defined PROXY option,
rechecksum the packet, and send it on it’s

It can inspect arriving packets, check for the
option, and decide if that packet should be

So What’s Left?

It doesn’t (yet) actually proxy the

Nor does it send an acknowledgement
of receipt to the upstream proxy.

There are also issues around ICMP error
messages, and what should be done
about them.


Once the code is “alpha” quality, I’ll
commit it to the swift CVS repository for
your collective viewing pleasure.

Until then, if you have questions or
suggestions, see me.