go-llm/docs/sandbox-setup.md

Sandbox Setup & Hardening Guide

Complete guide for setting up a Proxmox VE host to run isolated LXC sandbox containers for the go-llm sandbox package.

Table of Contents

1. Prerequisites
2. Proxmox Host Preparation
3. Network Setup
4. LXC Template Creation
5. SSH Key Setup
6. Configuration
7. Hardening Checklist
8. Monitoring & Maintenance
9. Troubleshooting

---

1. Prerequisites

Hardware/VM Requirements

Resource	Minimum	Recommended
CPU	4 cores	8+ cores
RAM	8 GB	16+ GB
Storage	100 GB SSD	250+ GB SSD
Network	1 NIC	2 NICs (mgmt + sandbox)

Software

• Proxmox VE 8.x (installation guide)
• During install, configure the management interface on vmbr0

---

2. Proxmox Host Preparation

Create Resource Pool

Scope sandbox containers to a dedicated resource pool to limit API token access:

pvesh create /pools --poolid sandbox-pool

Create API User and Token

# Create dedicated user
pveum useradd mort-sandbox@pve

# Create role with minimum required permissions
pveum roleadd SandboxAdmin -privs "VM.Allocate,VM.Clone,VM.Audit,VM.PowerMgmt,VM.Console,Datastore.AllocateSpace,Datastore.Audit"

# Grant role on the sandbox pool only
pveum aclmod /pool/sandbox-pool -user mort-sandbox@pve -role SandboxAdmin

# Grant access to the template storage
pveum aclmod /storage/local -user mort-sandbox@pve -role PVEDatastoreUser

# Create API token (privsep=0 means token inherits user's permissions)
pveum user token add mort-sandbox@pve sandbox-token --privsep=0

Save the output — it contains the token secret:

┌──────────┬──────────────────────────────────────────┐
│ key      │ value                                    │
╞══════════╪══════════════════════════════════════════╡
│ full-tokenid │ mort-sandbox@pve!sandbox-token       │
│ value    │ xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx     │
└──────────┴──────────────────────────────────────────┘

Store the secret securely (environment variable, secret manager, etc.). Never commit it to source control.

---

3. Network Setup

3.1 Create Isolated Bridge

Add to /etc/network/interfaces on the Proxmox host:

auto vmbr1
iface vmbr1 inet static
    address 10.99.0.1/16
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
    # NAT for optional internet access (controlled per-container by nftables)
    post-up   nft add table nat 2>/dev/null; true
    post-up   nft add chain nat postrouting { type nat hook postrouting priority 100 \; } 2>/dev/null; true
    post-up   nft add rule nat postrouting oifname "vmbr0" ip saddr 10.99.0.0/16 masquerade 2>/dev/null; true

Apply the configuration:

ifreload -a

3.2 Install and Configure DHCP

apt-get install -y dnsmasq

Create /etc/dnsmasq.d/sandbox.conf:

interface=vmbr1
bind-interfaces
dhcp-range=10.99.1.1,10.99.254.254,255.255.0.0,1h
dhcp-option=option:router,10.99.0.1
dhcp-option=option:dns-server,1.1.1.1,8.8.8.8

Restart dnsmasq:

systemctl restart dnsmasq
systemctl enable dnsmasq

3.3 Configure nftables Firewall

Create /etc/nftables.conf:

#!/usr/sbin/nft -f

flush ruleset

table inet sandbox {
    # Dynamic set of container IPs allowed internet access.
    # Populated/cleared by the sandbox manager via the Proxmox API.
    set internet_allowed {
        type ipv4_addr
    }

    chain forward {
        type filter hook forward priority 0; policy drop;

        # Allow established/related connections
        ct state established,related accept

        # Allow inter-bridge traffic (host ↔ containers via vmbr1)
        iifname "vmbr1" oifname "vmbr1" accept

        # Allow DNS for all containers (needed for apt)
        ip saddr 10.99.0.0/16 udp dport 53 accept
        ip saddr 10.99.0.0/16 tcp dport 53 accept

        # Allow HTTP/HTTPS only for containers in the internet_allowed set
        ip saddr @internet_allowed tcp dport { 80, 443 } accept

        # Rate limit: max 50 new connections per second per container
        ip saddr 10.99.0.0/16 ct state new limit rate over 50/second drop

        # Block everything else from containers
        ip saddr 10.99.0.0/16 drop

        # Allow host → containers (for SSH from the application)
        ip daddr 10.99.0.0/16 accept
    }

    chain input {
        type filter hook input priority 0; policy accept;

        # Block containers from accessing Proxmox management ports
        # (only SSH is allowed for the sandbox manager)
        iifname "vmbr1" ip daddr 10.99.0.1 tcp dport != 22 drop
    }
}

# NAT table for optional internet access
table nat {
    chain postrouting {
        type nat hook postrouting priority 100;
        oifname "vmbr0" ip saddr 10.99.0.0/16 masquerade
    }
}

Apply and persist:

nft -f /etc/nftables.conf
systemctl enable nftables

Verify:

nft list ruleset

3.4 Test Network Isolation

From a test container on vmbr1:

# Should work: DNS resolution
dig google.com

# Should be blocked: HTTP (not in internet_allowed set)
curl -s --connect-timeout 5 https://google.com && echo "FAIL: should be blocked" || echo "OK: blocked"

# Should be blocked: access to LAN
ping -c 1 -W 2 192.168.1.1 && echo "FAIL: LAN reachable" || echo "OK: LAN blocked"

# Should be blocked: access to Proxmox management
curl -s --connect-timeout 5 https://10.99.0.1:8006 && echo "FAIL: Proxmox reachable" || echo "OK: Proxmox blocked"

---

4. LXC Template Creation

4.1 Download Base Image

pveam update
pveam download local ubuntu-24.04-standard_24.04-1_amd64.tar.zst

4.2 Create Template Container

pct create 9000 local:vztmpl/ubuntu-24.04-standard_24.04-1_amd64.tar.zst \
    --hostname sandbox-template \
    --memory 1024 \
    --swap 0 \
    --cores 1 \
    --rootfs local-lvm:8 \
    --net0 name=eth0,bridge=vmbr1,ip=dhcp \
    --unprivileged 1 \
    --features nesting=0 \
    --ostype ubuntu \
    --ssh-public-keys /root/.ssh/mort_sandbox.pub \
    --pool sandbox-pool \
    --start 0

4.3 Install Base Packages

pct start 9000
pct exec 9000 -- bash -c '
    apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        python3 python3-pip python3-venv \
        nodejs npm \
        git curl wget jq \
        vim nano \
        htop tree \
        ca-certificates \
        openssh-server \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
'

4.4 Create Sandbox User

pct exec 9000 -- bash -c '
    # Create unprivileged sandbox user with sudo
    useradd -m -s /bin/bash -G sudo sandbox
    echo "sandbox ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/sandbox

    # Set up SSH access
    mkdir -p /home/sandbox/.ssh
    cp /root/.ssh/authorized_keys /home/sandbox/.ssh/
    chown -R sandbox:sandbox /home/sandbox/.ssh
    chmod 700 /home/sandbox/.ssh
    chmod 600 /home/sandbox/.ssh/authorized_keys

    # Create uploads directory
    mkdir -p /home/sandbox/uploads
    chown sandbox:sandbox /home/sandbox/uploads

    # Enable SSH
    systemctl enable ssh
'

4.5 Security Hardening

pct exec 9000 -- bash -c '
    # Process limits (prevent fork bombs)
    echo "* soft nproc 256" >> /etc/security/limits.conf
    echo "* hard nproc 512" >> /etc/security/limits.conf

    # Disable core dumps
    echo "* hard core 0" >> /etc/security/limits.conf

    # Disable unnecessary services
    systemctl disable systemd-resolved 2>/dev/null || true
    systemctl disable snapd 2>/dev/null || true
'

4.6 Convert to Template

pct stop 9000
pct template 9000

4.7 Verify Template

Clone and test manually:

pct clone 9000 9999 --hostname test-sandbox --full
pct start 9999
# Wait for DHCP, then SSH in
ssh sandbox@<container-ip>
# Run some commands, verify packages installed
sudo apt-get update
python3 --version
node --version
# Clean up
exit
pct stop 9999
pct destroy 9999

---

5. SSH Key Setup

5.1 Generate Key Pair

ssh-keygen -t ed25519 -f /etc/mort/sandbox_key -N "" -C "mort-sandbox"

5.2 Install Public Key in Template

This was done in step 4.2 with --ssh-public-keys. If you need to update it:

pct start 9000  # Only if template — you'll need to untemplate first
# Copy key
cat /etc/mort/sandbox_key.pub | pct exec 9000 -- tee /home/sandbox/.ssh/authorized_keys
pct exec 9000 -- chown sandbox:sandbox /home/sandbox/.ssh/authorized_keys
pct exec 9000 -- chmod 600 /home/sandbox/.ssh/authorized_keys
pct stop 9000
pct template 9000

5.3 Set Permissions

chmod 600 /etc/mort/sandbox_key
chmod 644 /etc/mort/sandbox_key.pub
# If running as a specific user:
chown mort:mort /etc/mort/sandbox_key /etc/mort/sandbox_key.pub

---

6. Configuration

Go Configuration

signer, _ := sandbox.LoadSSHKey("/etc/mort/sandbox_key")

mgr, _ := sandbox.NewManager(sandbox.Config{
    Proxmox: sandbox.ProxmoxConfig{
        BaseURL:            "https://proxmox.local:8006",
        TokenID:            "mort-sandbox@pve!sandbox-token",
        Secret:             os.Getenv("SANDBOX_PROXMOX_SECRET"),
        Node:               "pve",
        TemplateID:         9000,
        Pool:               "sandbox-pool",
        Bridge:             "vmbr1",
        InsecureSkipVerify: true, // Only for self-signed certs
    },
    SSH: sandbox.SSHConfig{
        Signer:         signer,
        User:           "sandbox",  // default
        ConnectTimeout: 10 * time.Second,  // default
        CommandTimeout: 60 * time.Second,  // default
    },
    Defaults: sandbox.ContainerConfig{
        CPUs:     1,
        MemoryMB: 1024,
        DiskGB:   8,
    },
})

Environment Variables

Variable	Description
SANDBOX_PROXMOX_SECRET	Proxmox API token secret
SANDBOX_SSH_KEY_PATH	Path to SSH private key (alternative to config)

---

7. Hardening Checklist

Run through this checklist after setup:

Container Isolation

• Containers are unprivileged (verify UID mapping in /etc/pve/lxc/<id>.conf)
• Nesting is disabled (features: nesting=0)
• Swap is disabled on containers (swap: 0)
• Resource pool scoping: API token can only touch sandbox-pool

Network Isolation

• vmbr1 has no physical ports (bridge-ports none)
• nftables rules loaded: nft list ruleset shows sandbox table
• nftables persists across reboots: systemctl is-enabled nftables
• Default-deny outbound for containers
• DNS (port 53) allowed for all containers
• HTTP/HTTPS only for containers in internet_allowed set
• Rate limiting active (50 conn/sec)
• Containers cannot reach Proxmox management (port 8006 blocked)

Security Profiles

• AppArmor profile active: lxc-container-default-cgns
• Process limits in /etc/security/limits.conf (nproc 256/512)
• Core dumps disabled
• Capability drops verified in container config

Functional Tests

• Fork bomb test: run :(){ :|:& };: in container → PID limit fires, container survives
• OOM test: allocate >1GB memory → container OOM-killed, host unaffected
• Network scan test: nmap from container → blocked by nftables
• Container escape test: attempt to mount host filesystem → denied
• LAN access test: ping LAN hosts → blocked
• Cross-container test: ping other sandbox containers → blocked
• Internet access test: HTTP without being in internet_allowed → blocked
• Internet access test: add to internet_allowed → HTTP works
• Cleanup test: destroy container → verify no orphan volumes

Operational Tests

• Clone template → container starts → SSH connects → commands work
• File upload/download via SFTP works
• Container destroy removes all resources
• Orphan cleanup: kill application mid-session, restart, verify cleanup

---

8. Monitoring & Maintenance

Log Rotation

Sandbox session logs should be rotated to prevent disk exhaustion. If using slog to a file:

# /etc/logrotate.d/sandbox
/var/log/sandbox/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
}

Storage Cleanup

Verify destroyed containers don't leave orphan volumes:

# List all LVM volumes in the sandbox storage
lvs | grep sandbox

# Compare with running containers
pct list | grep sandbox

Template Updates

Periodically update the template with latest packages:

# Un-template (creates a regular container from template)
# Note: you can't un-template directly; clone then replace
pct clone 9000 9001 --hostname template-update --full
pct start 9001
pct exec 9001 -- bash -c 'apt-get update && apt-get upgrade -y && apt-get clean'
pct stop 9001

# Destroy old template and create new one
pct destroy 9000
# Rename 9001 → 9000 (or update your config to use the new ID)
pct template 9001

Proxmox Host Updates

apt-get update && apt-get dist-upgrade -y
# Reboot if kernel was updated
# Verify nftables rules are still loaded after reboot
nft list ruleset

---

9. Troubleshooting

Container won't start

# Check task log
pct start <id>
# If error, check:
journalctl -u pve-container@<id> -n 50

# Common issues:
# - Storage full: check `df -h` and `lvs`
# - UID mapping issues: verify /etc/subuid and /etc/subgid

SSH connection refused

# Verify container is running
pct status <id>

# Check if SSH is running inside container
pct exec <id> -- systemctl status ssh

# Verify IP assignment
pct exec <id> -- ip addr show eth0

# Check DHCP leases
cat /var/lib/misc/dnsmasq.leases

Container has no internet (when it should)

# Verify container IP is in the internet_allowed set
nft list set inet sandbox internet_allowed

# Manually add for testing
nft add element inet sandbox internet_allowed { 10.99.1.5 }

# Verify NAT is working
nft list table nat

# Check if IP forwarding is enabled
cat /proc/sys/net/ipv4/ip_forward  # Should be 1

nftables rules lost after reboot

# Verify nftables is enabled
systemctl is-enabled nftables

# If rules are missing, reload
nft -f /etc/nftables.conf

# Make sure the config file is correct
nft -c -f /etc/nftables.conf  # Check syntax without applying

Orphaned containers

# List all containers in the sandbox pool
pvesh get /pools/sandbox-pool --output-format json | jq '.members[] | select(.type == "lxc")'

# Destroy orphans manually
pct stop <id> && pct destroy <id> --force --purge