Linux tproxy server in Java - Graham Edgecombe

29 December 2013

tproxy is a feature in Linux which allows an intermediate router to run a proxy server which can intercept and modify network traffic transparently (i.e. the end systems cannot tell that this has been done, as the source/destination IP addresses in the packets are not modified.) tproxy also works with IPv6 whereas non-transparent mechanisms such as the iptables REDIRECT target do not because of the lack of NAT support in the Linux IPv6 stack in older kernels.

Java’s standard library does not provide built-in support for using tproxy, and it has a few annoyances which make it tricky to implement.

The main operation required to enable tproxy on a socket is to set the IP_TRANSPARENT option with the setsockopt() system call. For example, the code to enable tproxy would look something like this in C:

int yes = 1;
if (setsockopt(fd, SOL_IP, IP_TRANSPARENT, &yes, sizeof(int)) < 0)
{
  perror("setsockopt");
  exit(1);
}

where fd is the file descriptor of the socket. The option must be set before the socket is bound with the bind() system call.

JNA is a library which makes it easy to call C functions from Java. Unlike JNI, you do not need to write any ‘glue’ code - you can convert the C signature to a Java signature, and JNA does all the hard work for you.

Here’s the JNA code which allows setsockopt() to be called from Java:

public interface CLibrary extends Library {
    public final CLibrary INSTANCE = (CLibrary) Native.loadLibrary("c", CLibrary.class);

    /* from /usr/include/bits/in.h */
    public final int SOL_IP = 0;
    public final int IP_TRANSPARENT = 19;

    /* from /usr/include/sys/socket.h */
    public int setsockopt(int socket, int level, int option_name, Pointer option_value, int option_len) throws LastErrorException;

    /* from /usr/include/string.h */
    public String strerror(int errnum);
}

Then setsockopt() can be called from Java like so:

IntByReference yes = new IntByReference(1);
try {
    /* option_len = sizeof(int) = 4 */
    CLibrary.INSTANCE.setsockopt(fd, CLibrary.SOL_IP, CLibrary.IP_TRANSPARENT, yes.getPointer(), 4);
} catch (LastErrorException ex) {
    throw new IOException("setsockopt: " + CLibrary.INSTANCE.strerror(ex.getErrorCode()));
}

So we just need to create a new java.net.Socket object, call setsockopt() and finally call bind() on the socket - easy, right? Unfortunately, it’s not quite so simple - creating a new Socket object in Java (in OpenJDK and the Oracle JVM) does not actually allocate a file descriptor. Instead, the file descriptor is allocated within Java’s bind() function itself - making it rather difficult to call setsockopt() at the appropriate point.

If you dig into the OpenJDK code, you can find the following native function which actually allocates the file descriptor (which I have simplified for readability):

/*
 * Class:     java_net_PlainSocketImpl
 * Method:    socketCreate
 * Signature: (Z)V */
JNIEXPORT void JNICALL
Java_java_net_PlainSocketImpl_socketCreate(JNIEnv *env, jobject this,
                                           jboolean stream) {
    jobject fdObj;
    int fd;

    fdObj = (*env)->GetObjectField(env, this, psi_fdID);
    if ((fd = JVM_Socket(domain, type, 0)) == JVM_IO_ERR) { /* calls the system's socket() function to allocate an fd */
        /* raise exception */
    }

    (*env)->SetIntField(env, fdObj, IO_fd_fdID, fd);
}

This is called from AbstractPlainSocketImpl’s create() method:

protected synchronized void create(boolean stream) throws IOException {
    fd = new FileDescriptor();
    socketCreate(stream); /* calls the native code from above */
}

In turn, the ServerSocket class itself calls the SocketImpl’s create() method within its own createImpl() method:

void createImpl() throws SocketException {
    try {
        impl.create(true); /* calls create() from above */
        created = true;
    } catch (IOException e) {
        throw new SocketException(e.getMessage());
    }
}

Until the socket has been bound or connected, the only way to trigger a call to createImpl() is via the bind() or connect() methods.

An astute reader might recognize the possible solution of calling createImpl() via reflection at this point to force the file descriptor to be allocated, which would then allow setsockopt() to be called before bind(). However, I have not managed to get this approach to work.

Instead, it turns out that the equivalent class to ServerSocket in NIO - ServerSocketChannel - allocates the file descriptor upon calling ServerSocketChannel.open(). Binding is done afterwards with a separate bind() method - which allows the setsockopt() call to be inserted in the correct place.

You can see this in the sun.nio.ch.ServerSocketChannelImpl class - the constructor calls Net.serverSocket():

ServerSocketChannelImpl(SelectorProvider sp) throws IOException {
    super(sp);
    this.fd =  Net.serverSocket(true);
    this.fdVal = IOUtil.fdVal(fd);
    this.state = ST_INUSE;
}

It’s fairly obvious to see that Net.serverSocket() allocates a file descriptor without digging into the native code which implements socket0() and setfdVal():

static FileDescriptor serverSocket(boolean stream) {
    return IOUtil.newFD(socket0(isIPv6Available(), stream, true));
}

public static FileDescriptor newFD(int i) {
    FileDescriptor fd = new FileDescriptor();
    setfdVal(fd, i);
    return fd;
}

The next problem is how to extract the integer file descriptor of a ServerSocketChannel. This can be done with reflection by reading the fd field within the sun.nio.ch.ServerSocketChannelImpl object. The fd field is a java.io.FileDescriptor. The FileDescriptor object itself contains the underlying file descriptor from the operating system as an integer in a field also called fd.

The following code implements the reflection technique as described above:

private static final Class<?> SERVER_SOCKET_CHANNEL_IMPL;
private static final Field SERVER_SOCKET_CHANNEL_FD;
private static final Field FD;

static {
    try {
        SERVER_SOCKET_CHANNEL_IMPL = Class.forName("sun.nio.ch.ServerSocketChannelImpl");

        SERVER_SOCKET_CHANNEL_FD = SERVER_SOCKET_CHANNEL_IMPL.getDeclaredField("fd");
        SERVER_SOCKET_CHANNEL_FD.setAccessible(true);

        FD = FileDescriptor.class.getDeclaredField("fd");
        FD.setAccessible(true);
    } catch (NoSuchFieldException | ClassNotFoundException ex) {
        throw new ExceptionInInitializerError(ex);
    }
}

public static int getFileDescriptor(ServerSocketChannel channel) throws IOException {
    try {
        FileDescriptor fd = (FileDescriptor) SERVER_SOCKET_CHANNEL_FD.get(channel);
        return FD.getInt(fd);
    } catch (IllegalAccessException ex) {
        throw new IOException(ex);
    }
}

This can be combined with the setsockopt() JNA code to create an openTproxyServerSocket() function:

public static ServerSocketChannel openTproxyServerSocket() throws IOException {
    ServerSocketChannel ch = ServerSocketChannel.open();
    int fd = getFileDescriptor(ch);

    IntByReference yes = new IntByReference(1);
    try {
        /* option_len = sizeof(int) = 4 */
        CLibrary.INSTANCE.setsockopt(fd, CLibrary.SOL_IP, CLibrary.IP_TRANSPARENT, yes.getPointer(), 4);
    } catch (LastErrorException ex) {
        throw new IOException("setsockopt: " + CLibrary.INSTANCE.strerror(ex.getErrorCode()));
    }

    return ch;
}

openTproxyServerSocket() could be used like so:

ServerSocketChannel ch = openTproxyServerSocket();
ch.bind(new InetSocketAddress(8443));
for (;;) {
    SocketChannel ch0 = ch.accept();
    /* do something with ch0 */
}

IP_TRANSPARENT can be set on a SocketChannel (as opposed to a ServerSocketChannel) in a completely analogous way - ServerSocketChannel and ServerSocketChannelImpl need to be replaced with SocketChannel and SocketChannelImpl respectively. The usage of the socket is the only part which is significantly different:

SocketChannel ch = openTproxySocket();
ch.bind(new InetSocketAddress("royal.gov.uk", 49152));   /* spoofed source IP address */
ch.connect(new InetSocketAddress("whitehouse.gov", 80)); /* destination IP address */

If you’re happy with using NIO, then that’s all the code you need. However, I was adding tproxy support to an existing program which used Java’s old-style I/O rather extensively - in particular, the SSLSocket class. Converting all the code to use NIO would be quite a lot of work - particularly as SSLEngine (the recommended way to use SSL with NIO in Java) is not a particularly friendly class.

It turns out that SocketChannel and ServerSocketChannel both have a socket() method which return an old-style Socket and ServerSocket respectively, which (supposedly) work as if you’d called new Socket() or new ServerSocket(), but in reality forward all the method calls to the corresponding NIO channel.

There is actually a long-standing bug I came across while doing this (it has been in the Java bug tracker since 2001). A Socket which is backed by a SocketChannel cannot read() and write() simultaneously - one call will block until the other completes. This is quite problematic in an application which is proxying data back and forth - consider the following example:

Proxy server opens connection to example.com:80.
Thread 1 in the proxy server calls read(). read() will not return until at least one byte of data is read.
Thread 2 in the proxy server calls write() with the HTTP request. write() dutifully waits for read() to finish. However, read() will never finish (ignoring a timeout, by which time it is too late anyway) because the HTTP server at example.com has not received a request to reply to.

To see why it happens, we need to dig into the SocketAdaptor source code - this is the class which extends Socket but forwards all of its calls to its corresponding SocketChannel. Its getInputStream() method returns a ChannelInputStream whose read() method does the following (if a timeout is not set, which is the default):

protected int read(ByteBuffer bb)
    throws IOException
{
    synchronized (sc.blockingLock()) {
        if (!sc.isBlocking())
            throw new IllegalBlockingModeException();
        if (timeout == 0)
            return sc.read(bb);

        /* code that won't be reached */
    }
}

SocketAdaptor’s getOutputStream() method returns an OutputStream whose write() method ultimately calls Channels.writeFully():

private static void writeFully(WritableByteChannel ch, ByteBuffer bb)
    throws IOException
{
    if (ch instanceof SelectableChannel) {
        SelectableChannel sc = (SelectableChannel)ch;
        synchronized (sc.blockingLock()) {
            if (!sc.isBlocking())
                throw new IllegalBlockingModeException();
            writeFullyImpl(ch, bb);
        }
    } else {
        writeFullyImpl(ch, bb);
    }
}

Both of these methods acquire the SocketChannel’s blockingLock() and then block. The lock is not released it until the operation has completed, which is the root cause of the bug.

To work around this bug, I created a class extending Socket (imaginatively called WorkaroundNioSocket). It overrides and passes through almost every method call to another Socket object given in the constructor - the socket obtained from SocketChannel.socket(). The only method which is not passed through directly is getOutputStream(), which wraps the underlying SocketChannel in a WritableByteChannel and returns Channels.newOutputStream() on the wrapped channel. WritableByteChannel does not extend SelectableChannel, so this prevents Channels.writeFully() from trying to obtain the blockingLock(), which allows simultaneous reads/writes to work again.

public final class WorkaroundNioSocket extends Socket {
    private final Socket socket;

    public WorkaroundNioSocket(SocketChannel channel) {
        this.socket = channel.socket();
    }

    @Override
    public OutputStream getOutputStream() throws IOException {
        final SocketChannel ch = socket.getChannel();
        return Channels.newOutputStream(new WritableByteChannel() {
            @Override
            public int write(ByteBuffer src) throws IOException {
                return ch.write(src);
            }

            @Override
            public boolean isOpen() {
                return ch.isOpen();
            }

            @Override
            public void close() throws IOException {
                ch.close();
            }
        });
    }

    /* pass through every other method */
    @Override
    public void connect(SocketAddress endpoint) throws IOException {
        socket.connect(endpoint);
    }

    /* etc */
}

Something similar also needs to be done for the ServerSocket - this time to ensure Sockets it accept()s are wrapped with WorkaroundNioSocket objects:

public final class WorkaroundNioServerSocket extends ServerSocket {
    private final ServerSocket socket;

    public WorkaroundNioServerSocket(ServerSocketChannel channel) throws IOException {
        this.socket = channel.socket();
    }

    @Override
    public Socket accept() throws IOException {
        Socket client = socket.accept();
        return new WorkaroundNioSocket(client.getChannel());
    }

    /* pass through every other method */
    @Override
    public void bind(SocketAddress endpoint) throws IOException {
        socket.bind(endpoint);
    }

    /* etc */
}

These wrapper classes can be used by replacing the ch.socket() call with either new WorkaroundNioSocket(ch) or new WorkaroundNioServerSocket(ch).

Bringing everything in this post together allows you to transparently proxy SSL traffic in Java on Linux, the goal I wanted to achieve in the project I’m working on:

ServerSocket server = new WorkaroundNioServerSocket(openTproxyServerSocket());
for (;;) {
    Socket socket1 = server.accept();
    SocketAddress src = socket1.getRemoteSocketAddress();
    SocketAddress dst = socket1.getLocalSocketAddress();

    Socket socket2 = new WorkaroundNioSocket(openTproxySocket());
    socket2.bind(src);
    socket2.connect(dst);

    SSLContext ctx = ...;
    SSLSocket sslSocket1 = (SSLSocket) ctx.getSocketFactory().createSocket(socket1, ...);
    sslSocket1.setUseClientMode(false);
    SSLSocket sslSocket2 = (SSLSocket) ctx.getSocketFactory().createSocket(socket2, ...);

    /* code to copy data from sslSocket1 -> sslSocket2 and vice-versa */
}