Say you have built something with an LLM. It works, but not well enough. The summarizer misses details that matter, or the agent keeps calling the wrong tool. What are our options?

Broadly, there are two. We can change the model itself, which is the fine-tuning family. Or we can change what we tell it, which is prompt engineering, though I’ll use the broader term text-based refinement here, since the interesting new methods go well beyond tweaking prompts by hand.

Changing the model (parametric learning)

An LLM’s knowledge lives in billions of numbers called parameters, or weights. Training means nudging those numbers, which is why this family of approaches is called parametric learning.

The first thought that comes to mind is supervised fine-tuning (SFT). Collect thousands of examples of the behavior you want, each one an input paired with the ideal output, and adjust the weights until the model imitates them reliably. It is apprenticeship by imitation: watch the master, copy the master.

The other option is reinforcement learning (RL), with methods like GRPO. Here we don’t show the model correct answers. We let it attempt the task, score each attempt, and shift the weights toward whatever scores well. Learning by trial, error and grades.

Both are powerful, but the costs are not as fixed as they sound. Full fine-tuning of a large model does need a big dataset and serious GPU time. Parameter-efficient techniques like LoRA change that math: they train small adapter layers on top of a frozen base model, which brings the data and compute requirements down dramatically (I wrote about LoRA adapters earlier). Many providers now offer fine-tuning as a hosted service too, so sitting behind an API no longer rules it out. RL still needs a reliable way to score attempts, and enough of them to learn from. The one thing that holds across the board is opacity: the improvement lands in the weights, billions of numbers that nobody can read.

Changing the instructions (text-based refinement)

The alternative is to keep the model frozen and improve the text around it: the prompts, the rules, the playbook the model follows.

If parametric learning is sending an employee away for months of retraining, text-based refinement is editing the employee handbook. Same person, sharper instructions.

Everyone already does this by hand. Try a prompt, watch it fail, reword it, try again. The interesting development is that this loop can now be automated, and the best automated versions learn from something richer than a score.

The score versus the explanation

Imagine you’re helping a new engineer get better at code reviews.

One approach is to score every review: 6 out of 10, 8 out of 10, 4 out of 10. After enough attempts, perhaps they slowly develop better instincts.

A more useful approach is to sit down with one review and say: “You caught the null check, but missed the race condition because you only followed the happy path. Next time, trace what happens when two requests update the same object.”

The score tells them how well they did. The explanation tells them how to improve.

Keep that distinction in mind. Parametric RL learns from scores. The most interesting text-based methods learn from explanations.

LLM-as-a-judge

Before anything can improve, something has to define what “better” means. An LLM-as-a-judge is simply an LLM used as an evaluator: given an output and a rubric, it returns a score, a critique, or both.

Note that a judge by itself improves nothing. It plays the role of a teacher grading an assignment, the red ink rather than the revision. The judge is also optional: feedback can come from unit tests, compiler errors, exact-match checks, or human comments.

TextGrad

Real LLM systems are rarely a single prompt. A research assistant might have one prompt that writes search queries, a retrieval step, and another prompt that composes the final answer. When the final answer is bad, which prompt is at fault?

TextGrad is an optimizer inspired by backpropagation, the algorithm used to train neural networks. It records how text flows through the pipeline, then carries criticism backward through it. If a judge says the answer lacks supporting evidence, TextGrad translates that into one critique for the answer-writing prompt and a different critique for the query-writing prompt. An LLM then revises each piece of text accordingly.

These “textual gradients” are natural-language suggestions, not numerical derivatives. The spirit is the same though: figure out which upstream component to blame, and route the criticism to it.

GEPA

GEPA (short for Genetic-Pareto) is a prompt optimizer that treats a failed attempt as a story worth reading, not just a number.

Say a retrieval agent gets a question wrong. A score of 0 tells us it failed. The execution trace, the record of what actually happened along the way, tells us much more:

• The first search found the right person but the wrong event.
• The second search merely rephrased the first.
• The retriever surfaced a useful source, but the final answer ignored it.

An LLM can read that trace and infer a general lesson: follow-up searches should use new entities discovered in earlier results instead of repeating the original query. GEPA writes that lesson back into the query-writing prompt and tries again.

The loop is roughly:

1. Run the current system on a few examples.
2. Reflect on its traces and feedback.
3. Rewrite one prompt in the system.
4. Test the new version and keep it if it helps.

This works for a single prompt, but it really shines in compound systems with several LLM calls, retrieval steps and tools, since GEPA can improve one module at a time instead of treating the whole pipeline as a black box.

The name describes the search strategy.

Genetic refers to evolution. GEPA maintains a population of prompt candidates. Reflection mutates one candidate, the new version inherits the rest of its parent’s instructions, and a merge step can combine improvements found along different lineages.

Pareto refers to selection. GEPA doesn’t only keep the candidate with the best average score. It also preserves candidates that are unusually good at something:

Candidate C can be discarded since A beats it on both examples. But A and B have different strengths, so GEPA keeps both alive. This prevents the optimizer from fixating on one early winner and getting stuck, and it lets useful specialist strategies survive long enough to contribute later.

Putting them side by side

Note the split in the “what changes” column. SFT and RL change the model. Everything else changes the words.

Which one should you use?

In the GEPA paper’s experiments across four tasks and two model families, GEPA outperformed both a GRPO fine-tuning setup and MIPROv2, an earlier automatic prompt optimizer, while sometimes using far fewer attempts. The Pareto strategy also beat greedily evolving only the single best prompt.

I’d treat those results as promising rather than universal. GEPA adds extra reflection calls, leans heavily on the quality of evaluator feedback, and was tested on a limited set of systems. With abundant data, updating weights may still be the better choice.

The honest answer is that it depends on more variables than data volume. The ones I find myself weighing:

1. Knowledge or behavior: If what’s missing is facts, especially facts that change often, retrieval (RAG) is usually the right tool and neither lever needs to move. Fine-tuning is better at teaching skills, style and format than at storing fresh information.
2. Rate of change: A fine-tuned model bakes in today’s behavior. If requirements or data shift every few weeks, retraining becomes a treadmill, while a prompt can be edited and shipped in an afternoon.
3. Auditability: An optimized prompt is a document. You can read it, diff it, and walk a reviewer through it. A weight update is a pile of numbers.
4. Economics: A long prompt is paid for on every single call, while fine-tuning moves that cost up front and makes each call cheaper. At high volume, the economics often decide ties.
5. Feedback richness: If failed attempts leave clues behind (compiler errors, missing documents, tool output, grading feedback), reflective methods like GEPA have material to work with. If all you get is a score, that advantage shrinks.

And whichever way you lean, build the eval first. Every technique in this post is steered by feedback, so the quality of your evaluation sets the ceiling on your optimization. Evals before opinions.

Conclusion

An execution trace isn’t just exhaust. It can be training data.

Parametric methods compress experience into weights you can’t read. Text-based methods compress it into instructions you can. You can open the optimized prompt, see the lesson the system learned, and decide whether you agree with it.

When an LLM can read its own attempts, understand the grader’s complaint, and rewrite the instruction that led it astray, prompt optimization starts to look less like wordsmithing and more like learning.

This is all for now, hopefully this was useful. I’d appreciate any feedback or comments.

References

1. Lakshya A. Agrawal et al., GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning, 2025.
2. DSPy GEPA documentation.
3. Mert Yuksekgonul et al., Optimizing generative AI by backpropagating language model feedback, 2025.
4. Zhihong Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (introduces GRPO), 2024.
5. Lianmin Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023.

Much of the conversation was about systems fundamentals: isolation, networking, resource limits, security, and what happened when things failed. Over time, the technology matured and those concerns became part of the platform. Most developers could simply assume that containers worked. Only specialized use cases continued to demand closer attention.

Today, working with AI agents gives me a similar feeling. We are once again exploring a new way of running software—and rediscovering why systems engineering matters.

The reason lies in how control shifts at runtime. Traditional applications follow paths written and reviewed in advance. Agents interpret goals, choose tools, interact with external systems, and decide what to do next at runtime. Some can operate for hours with limited supervision.

That changes the security boundary. We need to control not only where an agent runs, but also which tools it can use, what data it can access, where it can communicate, and how long it can continue. Policies must be explicit, credentials short-lived, and actions observable.

The familiar reliability problems return as well. Agents depend on APIs that time out, tools that partially succeed, and services that become unavailable. Long-running work requires checkpoints, durable state, bounded retries, and a safe way to resume without repeating actions.

Finally, autonomy makes resource management critical. An agent can consume compute, storage, model capacity, and external APIs for as long as it is allowed to run. And unlike a person, it does not get tired. It may continue retrying a broken operation or following an unproductive path indefinitely.

In many ways, agents are bringing us back to the questions we faced during the early days of containers. The problems are familiar, but the environment has changed. The model may be the most visible part of an agent, but systems engineering is what will allow agents to operate safely, reliably, and at scale.

1. What is LoRA, and why is it needed?

Full fine-tuning rewrites the encyclopedia and saves another giant copy. Three specialties can mean three complete model checkpoints.

LoRA (Low-Rank Adaptation) keeps the original model frozen and learns a slim packet of edits called an adapter. At each selected layer:

W is the frozen pretrained matrix. The adapter contains two much smaller trainable matrices, A and B. Rank r controls the narrow bottleneck between them.

LoRA works when the task-specific change fits inside this smaller subspace. It is effective often, but not guaranteed; rank and target layers still need evaluation.

2. Is training a LoRA adapter cheaper?

For a 4096 x 4096 matrix, full fine-tuning can update 16,777,216 parameters. A rank-16 adapter learns 131,072: 0.78% as many for that matrix.

Across 32 layers with LoRA on q_proj and v_proj, that illustrative rank-16 adapter is about 16 MiB in 16-bit storage. A 7-billion-parameter 16-bit base is about 14 GB: the saved adapter is roughly 800x smaller.

Training memory does not fall 800x because activations and the base weights remain. The saving comes from keeping gradients and optimizer state only for A and B.

There is no universal “training is N times cheaper” number. Sequence length, batch size, quantization, checkpointing, and target modules still dominate the real GPU bill.

3. Is serving LoRA cheaper?

The operational win is sharing one base. One hundred 16 MiB adapters add about 1.6 GB of artifacts; one hundred separate 14 GB checkpoints would be about 1.4 TB.

The implementation idea is small:

# Pseudocode: train independent adapters from one frozen base.
base = load_model("base", frozen=True)
for name, data in tasks:
    adapter = train_lora(base, data, rank=16)
    save_adapter(name, adapter)

# Serve one base with several named adapters.
server = load_model("base")
server.load_adapter("support")
server.load_adapter("finance")

for batch in requests:
    names = [route(request) for request in batch]
    responses = server.generate(batch, adapter_names=names)

Deployment Strategies in Practice

When putting LoRA into production, you have a few ways to manage how adapters interact with the base model:

• Switching: Activating one specific adapter for an entire request or a homogeneous batch.
• Mixed-Adapter Batching: Assigning different adapters to individual sequences within the same batch. Production-scale implementations typically use specialized serving engines such as vLLM or LoRAX.
• Composition: Mathematically blending multiple adapters together dynamically, rather than just routing requests between them.
• Merging: Permanently baking the adapter’s weights into the base model. This removes the separate adapter-layer inference overhead but destroys the ability to hot-swap adapters on the fly.

LoRA makes specialization dramatically smaller. Training still traverses the base, and serving still computes the base, but many specialists can share that one expensive foundation.

References

1. LoRA: Low-Rank Adaptation of Large Language Models
2. Hugging Face PEFT LoRA reference
3. PEFT checkpoint format

VXLAN is a Layer 2 overlay scheme on a Layer 3 network.

To understand VXLAN better, let’s first understand what’s subnetting and VLAN.

Let’s say we have a physical LAN where there are multiple hosts with IPs in 10.1.2.0/24 network. Each host can talk to the other host using a switch alone. Now, we want to group set of hosts and separate them from each other. What are our options?

IP Subnetting

The first thought that comes to my mind is IP subnetting. It can prevent two hosts in different subnets from talking to each other unless we explicitly allow it using a router.

Say now we have two subnets, 10.1.2.0/25 where the two ranges are 10.1.2.1 - 10.1.2.126 and 10.1.2.129 - 10.1.2.254. The broadcast address for first subnet is 10.1.2.127 and for second subnet it’s 10.1.2.255.

However, they still have the same layer 2 broadcast domain.

Consider the following example (in gif below) where even though the host (10.1.2.2) can’t ping the other hosts (10.1.2.130 and 10.1.2.134) but it can broadcast in the other subnet since the broadcast domain at layer 2 is same and switch will broadcast all the frames with destination FFFF.FFFF.FFFF to all the ports irrespective of layer 3 source/destination IPs.

The behavior is same in presence of router that allows the two subnets to talk to each other. Since switch operates at Layer 2.

VLAN

So how can we split the broadcast domains for the subnets? This is where VLAN comes in. VLAN can be used to separate the broadcast domains as well.

VLAN frames are tagged by switch based on what port they arrive at. VLAN header is 4 bytes long and is placed before the type field in ethernet frame. It contains a 12 bit VLAN Identifier (VID) that identifies the frame it belongs to.

In the example below, hosts (10.1.2.3, 10.1.2.2 and 10.1.2.135) are in the same vlan (vlan1) and the remaining hosts belong to a different vlan. As in the previous experiments, let’s ping the broadcast address (10.1.2.127) from the host 10.1.2.3. Note that in this case, the packet is only broadcasted to the hosts that belong to the same vlan irrespective of the subnet. Note that the host (10.1.2.35) will drop the packet since the destination belongs to a different subnet and we don’t have a default gateway set. However, hosts in the other vlan don’t even receive the icmp request.

Note that 12 bit identifier limits the maximum number of vlans you can have to 4096. However with recent influx of virtualized environments, this is not enough and VLAN is not friendly enough for large number of multi tenant topologies. Hence the proposal for VXLAN.

VXLAN

VXLAN stands for Virtual eXtensible Local Area Network, it’s a layer 2 in layer 3 overlay tunnel. More specifically an Ethernet in UDP tunnel. The idea of VXLAN is similar to VLAN, as in it provides logical separation of private networks.

Consider the network topology in the diagram below network A and B are part of the same private network separated by external network. A-P, B-Q use same subnet in the same physical network.

The requirement from underlay network (UDP) is that it should have ip connectivity between two networks, UDP port 4789 is reserved for VXLAN.

The endpoint of the tunnel, VTEP (Virtual Tunnel End Point) forms the control plane of VXLAN (overlay network). It maintains a mapping of internal MAC address to the outer IP address i.e., while sending a packet from A: 10.0.0.1 to B: 10.0.0.2. VTEP can learn

VMs in subnets (A and B in the above picture) are unaware of the VXLAN themselves, and they route the traffic as they normally would. Say, a VM (A, 10.0.0.1) wants to route traffic to another VM in same network (B, 10.0.0.2) located at a different physical server and network. It would send the MAC frame as it would if both were on the same physical network. VTEP on that host checks the MAC address of 10.0.0.2

Following packet trace shows the VXLAN packet wrapper in UDP outer packet. UDP forms the overlay network and inner nodes (with IP 10.0.0.1) don’t need to know about overlay network.

Conclusion

VXLAN is an important piece of puzzle when it comes to scaling ipv4 and providing network abstractions. I hope you have a slightly better understanding of where VXLAN belongs in plethora of networking technologies.

References:

[1] Packet trace for VXLAN - cloudshark
[2] RFC-7348: Virtual eXtensible Local Area Network (VXLAN)

You need linux headers to be able to compile new module.

To make sure you have linux headers, see if the following path exists:

$ ls /usr/src/linux-headers-`uname -r`
arch   crypto         drivers   fs       init  Kbuild   kernel  
...

If the above directory is not present, you can install the headers using:

sudo apt-get update && sudo apt-get install linux-headers-$(uname -r)

Once the headers are installed you are good to go.

Writing hello module

To start off, create the following file in any directory.

// hello.c
#include <linux/module.h>

MODULE_LICENSE("Dual BSD/GPL");

Create the following Makefile to compile your module.

obj-m := hello.o
KDIR := /usr/src/linux-headers-3.2.0-23-generic

all:
	make -C $(KDIR) M=$(PWD) modules
clean:
	make -C $(KDIR) M=$(PWD) clean

After running make, you should see following:

# Check that *.ko is created.
vagrant@precise64:~$ ls
hello.c  hello.ko  hello.mod.c  hello.mod.o  hello.o  Makefile  
...
# Install the module.
vagrant@precise64:~$ sudo insmod hello.ko
# Confirm the it's installed.
vagrant@precise64:~$ lsmod | grep hello
hello                   8217  0
# Remove the module.
vagrant@precise64:~$ sudo rmmod hello
# Confirm that it's removed.
vagrant@precise64:~$ lsmod | grep hello

Expanding further, each module needs to have an init and exit method.

#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>

MODULE_LICENSE("Dual BSD/GPL");

static int hello_init(void) {
  printk("<1> Hola Mundo!\n");
  return 0;
}

static void hello_exit(void) {
  printk("<1>Hasta Luego\n");
}

module_init(hello_init);
module_exit(hello_exit);

Again, use inmod to install the command and you should see Hola Mundo! in dmesg. printk is important here; as kernel will print it to the syslog. Pretty useful in understanding other drivers.

How do I tell if a device is in promiscuous mode?

tl;dr: Kernel tracks promiscuous mode using flags on the device. For promiscuous mode, IFF_PROMISC, 0x100 should be set.

For a given interface, check the flags to see if the promiscuous bit is set.

$ cat /sys/devices/virtual/net/veth0/flags
0x1303  # 0001 001[1] 0000 0011   # device is in promiscuous mode.

$ cat /sys/devices/virtual/net/br0/flags
0x1003  # 0001 000[0] 0000 0011  # device is not in promiscuous mode.

Here’s a quick python script to test promiscuous mode for all interfaces:

Problem with existing tools

Figuring out if a given network device is in promiscuous mode using tools like iproute2 or netstat can be trickier than you’d think.

At first glance, you’d think iproute2 or netstat -i command should tell you if the device is in promiscuous mode but that’s not always the case.

We’ll consider two examples here, first to show the case where it works as expected and second to show where it doesn’t.

A word on netstat:

In netstat command, flag P is used to display if the interface is in promiscuous mode. However, P is also used for point to point connection. You can verify from the net-tools code here

Example 1: When it works

Following example, sets the promiscuous mode on using the iproute2 and netstat -i command. You can verify it using the iproute2 command and kernel logs.

Verify that promiscuous mode is not enabled.

vagrant@precise64:~$ sudo ip link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 
          qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:88:0c:a6 brd ff:ff:ff:ff:ff:ff

vagrant@precise64:~$ sudo netstat -i eth0
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR  TX-OK TX-ERR TX-DRP TX-OVR  Flg
eth0    1500 0    28880  0      0      0      17050    0      0      0   BMRU

Enable the promiscuous mode.

vagrant@precise64:~$ sudo ip link set eth0 promisc on

Check if promiscuous mode is enabled (see PROMISC) using iproute2

vagrant@precise64:~$ sudo ip link show eth0
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 \
          qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:88:0c:a6 brd ff:ff:ff:ff:ff:ff

vagrant@precise64:~$ sudo netstat -i eth0
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR  TX-OK TX-ERR TX-DRP TX-OVR  Flg
eth0    1500 0    28880  0      0      0      17050    0      0      0   BMPRU

Let’s check the kernel log messages, as logged in __dev_set_promiscuity whenever a device is added/removed to/from promiscuous mode.

vagrant@precise64:~$ grep -r 'promiscuous' /var/log/kern.log
precise64 kernel: [44441.470885] device eth0 entered promiscuous mode

Example 2: When it doesn’t work

Consider this for example, adding an interface to bridge; set the promiscuous mode on for that interface. Check out the post Linux Bridge - how it works to learn more about bridge.

Following example creates a bridge, a veth pair and adds one end of the veth pair to bridge. According to br_add_if, promiscuous mode is turned on for the interface.

Create a new interface and add it to a bridge

vagrant@precise64:~$ sudo brctl addbr br0
vagrant@precise64:~$ sudo ip link add veth0 type veth peer name vpeer0
vagrant@precise64:~$ sudo brctl addif br0 veth0

Check if promiscuous mode is enabled using iproute2

vagrant@precise64:~$ sudo ip -d link show veth0
9: veth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master \
 br0 state DOWN qlen 1000
    link/ether 12:30:e3:6b:42:2d brd ff:ff:ff:ff:ff:ff
    veth

vagrant@precise64:~$ sudo netstat -ian veth0
Kernel Interface table
Iface  MTU Met RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
veth0   1500 0   0      0      0 0             0      0      0      0 BM
vpeer0  1500 0   0      0      0 0             0      0      0      0 BM

It doesn’t show the device to be in promiscuous mode as PROMISC is not set and the flag P is not present in netstat

Let’s check kernel logs and see if the device was actually put into promiscuous mode.

vagrant@precise64:~$ grep -r 'promiscuous' /var/log/kern.log
precise64 kernel: [43656.288050] device veth0 entered promiscuous mode

As expected, the device was in fact moved to promiscuous mode but iproute2 doesn’t show it in promiscuous mode.

References

1. https://lists.gt.net/linux/kernel/178148

As Anatomy of a Linux bridge puts it, bridge mainly consists of four major components:

• Set of network ports (or interfaces): used to forward traffic between end switches to other hosts in the network.
• A control plane: used to run Spanning Tree Protocol (STP) that calculates minimum spanning tree, preventing loops from crashing the network.
• A forwarding plane: used to process incoming input frames from the ports, forward them to the network port by making a forwarding decision based on the MAC learning database.
• MAC learning database: used to keep track of the host locations in the LAN.

For each unicast mac address, bridge maintains a mac learning database to decide which ports to forward based on MAC addresses, and if it can’t find an entry for a given mac address, it will broadcast the frame to all ports except the one where it received the frame from.

There are three main configuration subsystems to do bridges:

• ioctl: This interface is used to create/destroy bridges and add/remove interfaces to/from a bridge.
• sysfs: Management of bridge and port specific parameters.
• netlink: Asynchronous queue based communication that uses AF_NETLINK address family, can also be used to interact with bridge.

In this article, we only talk about ioctl.

Creating a bridge

Bridge can be created using ioctl command SIOCBRADDBR; as can be seen by brctl utility provided by bridge-utils.

vagrant@precise64:~$ sudo strace brctl addbr br1
execve("/sbin/brctl", ["brctl", "addbr", "br1"], [/* 16 vars */]) = 0
...
ioctl(3, SIOCBRADDBR, 0x7fff2eae9966)   = 0
...

Note that there is no device at this point to handle the ioctl command, so the ioctl command is handled by a stub method: br_ioctl_deviceless_stub, which in turn calls br_add_bridge. This method calls alloc_netdev, which is a macro that eventually calls alloc_netdev_mqs.

br_ioctl_deviceless_stub
  |- br_add_bridge
      |- alloc_netdev
           |- alloc_netdev_mqs  // creates the network device
              |- br_dev_setup // sets br_dev_ioctl handler               

alloc_netdev also initializes the new netdevice using the br_dev_setup. This also includes setting up the bridge specific ioctl handler. If you look at the handler code, it handles ioctl command to add/delete interfaces.

int br_dev_ioctl(struct net_device *dev, struct ifreq *rq, int cmd) {
    ...
    switch(cmd) {
	case SIOCBRADDIF:
	case SIOCBRDELIF:
		return add_del_if(br, rq->ifr_ifindex, cmd == SIOCBRADDIF);
    ...
    }
    ..
}

Adding an interface

As it can be seen in br_dev_ioctl, bridge can be created using ioctl command SIOCBRADDIF. To confirm:

vagrant@precise64:~$ sudo strace brctl addif br0 veth0
execve("/sbin/brctl", ["brctl", "addif", "br0", "veth0"], [/* 16 vars */]) = 0
...
# gets the index number of virtual ethernet device.
ioctl(4, SIOCGIFINDEX, {ifr_name="veth0", ifr_index=5}) = 0
close(4)
 # add the interface to bridge.
ioctl(3, SIOCBRADDIF, 0x7fff75bfe5f0)   = 0
...

br_add_if

The br_add_if method creates and sets up the new interface/port for the bridge by allocating a new net_bridge_port object. The object initialization is particularly interesting, as it sets the interface to receive all traffic, adds the network interface address for the new interface to the forwarding database as the local entry and attaches the interface as the slave to the bridge device.

/* Truncated version */
int br_add_if(struct net_bridge *br, struct net_device *dev)
{
	struct net_bridge_port *p;
	/* Don't allow bridging non-ethernet like devices */
    ...
	/* No bridging of bridges */
    ...
	p = new_nbp(br, dev);
    ...
	call_netdevice_notifiers(NETDEV_JOIN, dev);
	err = dev_set_promiscuity(dev, 1);
	err = kobject_init_and_add(&p->kobj, &brport_ktype, &(dev->dev.kobj),
				   SYSFS_BRIDGE_PORT_ATTR);
    ...
	err = netdev_rx_handler_register(dev, br_handle_frame, p);
    /* Make entry in forwarding database*/
	if (br_fdb_insert(br, p, dev->dev_addr, 0))
		...
    ...
}

Some things worth noting in br_add_if:

• Only ethernet like devices can be added to bridge, as bridge is a layer 2 device.
• Bridges cannot be added to a bridge.
• New interface is set to promiscuous mode: dev_set_promiscuity(dev, 1)

The promiscuous mode can be confirmed from kernel logs.

vagrant@precise64:~$ grep -r 'promiscuous' /var/log/kern.log
precise64 kernel: [ 5185.751666] device veth0 entered promiscuous mode

Finally, br_add_if method calls netdev_rx_handler_register, that sets the rx_handler of the interface to br_handle_frame

After this method finishes, you have an interface (or port) in bridge.

Frame Processing

Frame processing starts with device-independent network code, in __netif_receive_skb which calls the rx_handler of the interface, that was set to br_handle_frame at the time of adding the interface to bridge.

The br_handle_frame does the initial processing and any address with prefix 01-80-C2-00-00 is a control plane address, that may need special processing. From the comments in br_handle_frame:

    /*
    * See IEEE 802.1D Table 7-10 Reserved addresses
    *
    * Assignment		 		Value
    * Bridge Group Address		01-80-C2-00-00-00
    * (MAC Control) 802.3		01-80-C2-00-00-01
    * (Link Aggregation) 802.3	        01-80-C2-00-00-02
    * 802.1X PAE address		01-80-C2-00-00-03
    *
    * 802.1AB LLDP 		01-80-C2-00-00-0E
    *
    * Others reserved for future standardization
    */

In the method, note that stp messages are either passed to upper layers or forwarded if STP is enabled on the bridge or disabled respectively. Finally if a forwarding decision is made, the packet is passed to br_handle_frame_finish, where the actual forwarding happens.

Here’s the highly truncated version of br_handle_frame_finish:

/* note: already called with rcu_read_lock */
int br_handle_frame_finish(struct sk_buff *skb)
{
    struct net_bridge_port *p = br_port_get_rcu(skb->dev);
    ...
	/* insert into forwarding database after filtering to avoid spoofing */
	br = p->br;
	br_fdb_update(br, p, eth_hdr(skb)->h_source, vid);
	if (p->state == BR_STATE_LEARNING)
		goto drop;
	/* The packet skb2 goes to the local host (NULL to skip). */
	skb2 = NULL;
	if (br->dev->flags & IFF_PROMISC)
		skb2 = skb;
	dst = NULL;
	if (is_broadcast_ether_addr(dest))
		skb2 = skb;
	else if (is_multicast_ether_addr(dest)) {
        ...
	} else if ((dst = __br_fdb_get(br, dest, vid)) &&
			dst->is_local) {
		skb2 = skb;
		/* Do not forward the packet since it's local. */
		skb = NULL;
	}
	if (skb) {
		if (dst) {
			br_forward(dst->dst, skb, skb2);
		} else
			br_flood_forward(br, skb, skb2);
	}
	if (skb2)
		return br_pass_frame_up(skb2);
out:
	return 0;
    ...
}

As you can see in above snippet of br_handle_frame_finish,

• An entry in forwarding database is updated for the source of the frame.
• (not in the above snippet) If the destination address is a multicast address, and if the multicast is disabled, the packet is dropped. Or else message is received using br_multicast_rcv
• Now if the promiscuous mode is on, packet will be delivered locally, irrespective of the destination.
• For a unicast address, we try to determine the port using the forwarding database (__br_fdb_get).
• If the destination is local, then skb is set to null i.e., packet will not be forwarded.
• If the destination is not local, then based on if we found an entry in forwarding database, either the frame is forwarded (br_forward) or flooded to all ports (br_flood_forward).
• Later, packet is delivered locally (br_pass_frame_up) if needed (based on either the current host being the destination or the net device being in promiscuous mode).

br_forward method either clones and then deliver (if it is also to be delivered locally, by calling deliver_clone), or directly forwards the message to the intended destination interface by calling __br_forward.

bt_flood_forward forwards the frame on each interface by iterating through the list in br_flood method.

Conclusion

Bridges can be used to create various different network topologies and it’s important to understand how they work. I have seen bridges being used with containers where they are used to provide networking in network namespaces along with veth devices. In fact the default networking in docker is provided using bridge.

This is all for now, hopefully this was useful. This was mainly based on the excellent paper Anatomy of a Linux bridge and my own reading of linux kernel code. I’d appreciate any feedback, or comments.

Ah, the wonderful world of bridges.

Update July 12, 2017: Added third way to communicate with bridge. Thanks to @vbernat from comments.

References:

Handwritten notes

[1] Anatomy of a Linux bridge
[2] Understanding Linux Networking Internals - Christian Benvenuti
[3] Linux kernel v3.10.105 source code

References:

[1] Anatomy of a Linux bridge
[2] Understanding Linux Networking Internals - Christian Benvenuti

Netfilter and IPTables

Netfilter provides a set of hooks inside Linux kernel networking stack which allows one to take action at different phases in routing process.

iptables is a userland tool to interact with netfilter in kernel. IPTables and netfilter together facilitate in enforcing different packet-filtering rules and setting up a basic firewall.

To check if you have iptables enabled:

$ lsmod | grep ip_tables
ip_tables              27115  3 iptable_filter,iptable_mangle,iptable_nat

Packet Traversal, Tables and Chains

A packet goes through several different routines and there are hooks available that allow you to modify the behavior.

There are primarily 3 different tables which allow you to define rules and where those rules get applied depends on the chain.

Consider the following diagram showing packet traversal flow:

According to above diagram, a packet goes through following chains:

1. PREROUTING: A packet goes through this chain before any routing decision is made.
2. FORWARD: If the message is not intended for local machine; it can be either dropped or routed to other machine based on the policy.
3. INPUT: If the packet is targeted for local machine, it passes through the INPUT chain.
4. OUTPUT: Any packet originating from local machine goes through the OUTPUT chain.
5. POSTROUTING: All outbound packet goes through post routing state.

Let’s go though the packet flow. When a new packet arrives:

• It first goes through the PREROUTING chain; where you can change the TCP headers, setup nat rules, etc; before passing it on for the Routing. Routing decides if the packet is targeted for current system based on IP headers.
• If it is not meant for the current system, packet passes through the FORWARD chain where one can decide to either forward the packet or drop it based on the set policy. If the forwarding is allowed, the packet is passed on to the POSTROUTING chain, otherwise it’s dropped or rejected.
• If the packet is meant for the current system, it is passed to the INPUT chain where packets can be filtered, header can be modified, etc and once rules in this chain are applied, packet is passed on to the local machine for processing.
• Any new packet from local machine passes through OUTPUT chain, where packet can be modified before the routing decision is made. Once the routing decision is made, packet goes through the POSTROUTING chain.
• POSTROUTING change can modify the headers, filter the packets before packet is finally put onto the network.

I loosely talked about filters, modifying headers during the traversal flow of packet. To make it more concrete, let’s look at what tables are available and how can they be used. As mentioned above, there are primarily 3 tables available where you can define you rules and these rules are executed at different chains. These three tables are: nat, mangle and filter.

Not all tables are available in all chains, as you can see filter table is not available in PREROUTING and POSTROUTING chains.

So a rules needs to have a chain and a table. For example, an iptable rule looks something like this:

sudo iptables -t nat -A PREROUTING -p tcp --dport 22 -j DNAT \
    --to-destination 172.22.212.30:222

where table is nat and chain is PREROUTING.

The 3 tables (-t flag) and their functions:

1. filter: This table, as the name suggests, filters the packets based on the rules. This is the default table and will be used if a table is not specified. It is only present in INPUT, OUTPUT and FORWARD chain. Based on target value, packets can be dropped, accepted, and returned.
2. nat: This allows altering the packets destination or source address. This is normally used to modify source and destination headers to allow internal ips to talk to outside networks. Normal use-case involves changing the destination IP address of the packet, also known as DNAT, in the PREROUTING chain to that of local machine so that routing can route it to INPUT chain. And conversely, changing the source address of the packet, also known as SNAT, in the POSTROUTING chain to that of the router or an ip known to destination routing.
3. mangle: Mangle can be used to modify certain properties of the packet like TOS, TTL, MTU, etc.

Targets and Jumps (-j flag)

Each rule has a predicate and a possible action which is called a target. For a detailed list of possible target and jump values, refer: iptables - targets and Jumps. Some of the most common ones are:

1. ACCEPT: This can be used to allow traffic based on the other specifications in the rule.
2. DROP: This can be used to drop the messages. No further processing is done; and no further information will be sent back to sender.
3. REJECT: This is similar to DROP but it also sends back an error message back to host sending the packet.
4. SNAT: It allows one to modify source ip address in the packet. It’s only valid in POSTROUTING chain and nat table.
5. DNAT: It allows one to modify destination ip address in the packet. It’s only valid in PREROUTING chaing and nat table.
6. MASQUERADE: This is same as SNAT but it doesn’t require a --to-source option. I.e., this is preferable if you don’t know the external ip during the rule creation, as it may be retrieved later using DHCP.

Some CLI examples

This post was an attempt to understand basics of iptables, rather than a specific example. There are some good examples online like this nat tutorial

To drop traffic from a certain IP address

We can use filter table (which is the default) and add the rule in the INPUT chain.

# To create the rule
$ sudo iptables -A INPUT -s 192.12.33.12 -j REJECT

# To list the rule
$ sudo iptables -L OUTPUT -n
Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination
1    REJECT     all  --  0.0.0.0/0            192.12.33.12       \
                                     reject-with icmp-port-unreachable

# To delete the rule
$ sudo iptables -D OUTPUT -1

To drop all traffic to a certain IP address

We can use filter table (which is the default) and add the following rule in the OUTPUT chain.

sudo iptables -t filter -d 192.12.33.12 -A OUTPUT -j REJECT

For more complex examples, please refer to this: nat tutorial

This is it for now.

References:

[1] Taming the Wild Netfilter - linuxjournal.com
[2] Netfilter Architecture - netfilter.org

XINETD

It’s basically a daemon that listens for network requests and services them by spawning more processes.

The master configuration for xinetd lives in /etc/xinetd.conf. Each service managed by xinetd has a configuration file in /etc/xinetd.d/.

Each network service is listed in /etc/services that xinetd could potentially manage.

Let’s look at an example from one of the services in /etc/xinetd.d/ to see how it works:

An echo service

This was a default service that was present on my RHEL6 box. There were lots of settings in this file which were basically commented out. Most of them are self explanatory, so I have omitted them for brevity.

$ sudo cat /etc/xinetd.d/echo-stream
# This is the configuration for the tcp/stream echo service.

service echo
{
# This is for quick on or off of the service
	disable		= yes

# The next attributes are mandatory for all services
	id		= echo-stream
	type		= INTERNAL
	wait		= no
	socket_type	= stream
#	protocol	=  socket type is usually enough

}

echo service simply provides an echo service (duh). But what port does it listen to? The port can be checked in /etc/services file, search for echo in file, and on my machine it had an entry that looked like this:

$ sudo cat /etc/services | grep echo
echo            7/tcp

If you try to connect to this port; the connection will fail since the disabled flag is set to yes in the above configuration file.

$ telnet 172.22.210.126 7
Trying 172.22.210.126...
telnet: connect to address 172.22.210.126: Connection refused
telnet: Unable to connect to remote host

Let’s enable the service by setting disable = no in /etc/xinetd.d/echo-stream. In addition, you’d need to restart the xinetd service.

$ sudo service xinetd restart
Stopping xinetd:                                           [  OK  ]
Starting xinetd:                                           [  OK  ]

Now again, let’s try to connect to service.

$ telnet 172.22.210.126 7
Trying 172.22.210.126...
Connected to angoyal-ld2.linkedin.biz.
Escape character is '^]'.
hola  <---- I said hola to Server.
hola  ----> Server said hola back.
^]
telnet> q
Connection closed.

Sweet.

You can use xinetd to run your own network service and have full control. I have some ideas which I’ll document if they work.

So long.