go-llm/docs/sandbox-setup.md

# Sandbox Setup & Hardening Guide

Complete guide for setting up a Proxmox VE host to run isolated LXC sandbox containers for the go-llm sandbox package.

## Table of Contents

1. [Prerequisites](#1-prerequisites)
2. [Proxmox Host Preparation](#2-proxmox-host-preparation)
3. [Network Setup](#3-network-setup)
4. [LXC Template Creation](#4-lxc-template-creation)
5. [SSH Key Setup](#5-ssh-key-setup)
6. [Configuration](#6-configuration)
7. [Hardening Checklist](#7-hardening-checklist)
8. [Monitoring & Maintenance](#8-monitoring--maintenance)
9. [Troubleshooting](#9-troubleshooting)

---

## 1. Prerequisites

### Hardware/VM Requirements

| Resource | Minimum | Recommended |
|----------|---------|-------------|
| CPU | 4 cores | 8+ cores |
| RAM | 8 GB | 16+ GB |
| Storage | 100 GB SSD | 250+ GB SSD |
| Network | 1 NIC | 2 NICs (mgmt + sandbox) |

### Software

- Proxmox VE 8.x ([installation guide](https://pve.proxmox.com/wiki/Installation))
- During install, configure the management interface on `vmbr0`

---

## 2. Proxmox Host Preparation

### Create Resource Pool

Scope sandbox containers to a dedicated resource pool to limit API token access:

```bash
pvesh create /pools --poolid sandbox-pool
```

### Create API User and Token

```bash
# Create dedicated user
pveum useradd mort-sandbox@pve

# Create role with minimum required permissions
pveum roleadd SandboxAdmin -privs "VM.Allocate,VM.Clone,VM.Audit,VM.PowerMgmt,VM.Console,Datastore.AllocateSpace,Datastore.Audit"

# Grant role on the sandbox pool only
pveum aclmod /pool/sandbox-pool -user mort-sandbox@pve -role SandboxAdmin

# Grant access to the template storage
pveum aclmod /storage/local -user mort-sandbox@pve -role PVEDatastoreUser

# Create API token (privsep=0 means token inherits user's permissions)
pveum user token add mort-sandbox@pve sandbox-token --privsep=0
```

Save the output — it contains the token secret:
```
┌──────────┬──────────────────────────────────────────┐
│ key      │ value                                    │
╞══════════╪══════════════════════════════════════════╡
│ full-tokenid │ mort-sandbox@pve!sandbox-token       │
│ value    │ xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx     │
└──────────┴──────────────────────────────────────────┘
```

Store the secret securely (environment variable, secret manager, etc.). Never commit it to source control.

---

## 3. Network Setup

### 3.1 Create Isolated Bridge

Add to `/etc/network/interfaces` on the Proxmox host:

```
auto vmbr1
iface vmbr1 inet static
    address 10.99.0.1/16
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
    # NAT for optional internet access (controlled per-container by nftables)
    post-up   nft add table nat 2>/dev/null; true
    post-up   nft add chain nat postrouting { type nat hook postrouting priority 100 \; } 2>/dev/null; true
    post-up   nft add rule nat postrouting oifname "vmbr0" ip saddr 10.99.0.0/16 masquerade 2>/dev/null; true
```

Apply the configuration:
```bash
ifreload -a
```

### 3.2 Install and Configure DHCP

```bash
apt-get install -y dnsmasq
```

Create `/etc/dnsmasq.d/sandbox.conf`:
```
interface=vmbr1
bind-interfaces
dhcp-range=10.99.1.1,10.99.254.254,255.255.0.0,1h
dhcp-option=option:router,10.99.0.1
dhcp-option=option:dns-server,1.1.1.1,8.8.8.8
```

Restart dnsmasq:
```bash
systemctl restart dnsmasq
systemctl enable dnsmasq
```

### 3.3 Configure nftables Firewall

Create `/etc/nftables.conf`:

```nft
#!/usr/sbin/nft -f

flush ruleset

table inet sandbox {
    # Dynamic set of container IPs allowed internet access.
    # Populated/cleared by the sandbox manager via the Proxmox API.
    set internet_allowed {
        type ipv4_addr
    }

    chain forward {
        type filter hook forward priority 0; policy drop;

        # Allow established/related connections
        ct state established,related accept

        # Allow inter-bridge traffic (host ↔ containers via vmbr1)
        iifname "vmbr1" oifname "vmbr1" accept

        # Allow DNS for all containers (needed for apt)
        ip saddr 10.99.0.0/16 udp dport 53 accept
        ip saddr 10.99.0.0/16 tcp dport 53 accept

        # Allow HTTP/HTTPS only for containers in the internet_allowed set
        ip saddr @internet_allowed tcp dport { 80, 443 } accept

        # Rate limit: max 50 new connections per second per container
        ip saddr 10.99.0.0/16 ct state new limit rate over 50/second drop

        # Block everything else from containers
        ip saddr 10.99.0.0/16 drop

        # Allow host → containers (for SSH from the application)
        ip daddr 10.99.0.0/16 accept
    }

    chain input {
        type filter hook input priority 0; policy accept;

        # Block containers from accessing Proxmox management ports
        # (only SSH is allowed for the sandbox manager)
        iifname "vmbr1" ip daddr 10.99.0.1 tcp dport != 22 drop
    }
}

# NAT table for optional internet access
table nat {
    chain postrouting {
        type nat hook postrouting priority 100;
        oifname "vmbr0" ip saddr 10.99.0.0/16 masquerade
    }
}
```

Apply and persist:
```bash
nft -f /etc/nftables.conf
systemctl enable nftables
```

Verify:
```bash
nft list ruleset
```

### 3.4 Test Network Isolation

From a test container on `vmbr1`:
```bash
# Should work: DNS resolution
dig google.com

# Should be blocked: HTTP (not in internet_allowed set)
curl -s --connect-timeout 5 https://google.com && echo "FAIL: should be blocked" || echo "OK: blocked"

# Should be blocked: access to LAN
ping -c 1 -W 2 192.168.1.1 && echo "FAIL: LAN reachable" || echo "OK: LAN blocked"

# Should be blocked: access to Proxmox management
curl -s --connect-timeout 5 https://10.99.0.1:8006 && echo "FAIL: Proxmox reachable" || echo "OK: Proxmox blocked"
```

---

## 4. LXC Template Creation

### 4.1 Download Base Image

```bash
pveam update
pveam download local ubuntu-24.04-standard_24.04-1_amd64.tar.zst
```

### 4.2 Create Template Container

```bash
pct create 9000 local:vztmpl/ubuntu-24.04-standard_24.04-1_amd64.tar.zst \
    --hostname sandbox-template \
    --memory 1024 \
    --swap 0 \
    --cores 1 \
    --rootfs local-lvm:8 \
    --net0 name=eth0,bridge=vmbr1,ip=dhcp \
    --unprivileged 1 \
    --features nesting=0 \
    --ostype ubuntu \
    --ssh-public-keys /root/.ssh/mort_sandbox.pub \
    --pool sandbox-pool \
    --start 0
```

### 4.3 Install Base Packages

```bash
pct start 9000
pct exec 9000 -- bash -c '
    apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        python3 python3-pip python3-venv \
        nodejs npm \
        git curl wget jq \
        vim nano \
        htop tree \
        ca-certificates \
        openssh-server \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
'
```

### 4.4 Create Sandbox User

```bash
pct exec 9000 -- bash -c '
    # Create unprivileged sandbox user with sudo
    useradd -m -s /bin/bash -G sudo sandbox
    echo "sandbox ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/sandbox

    # Set up SSH access
    mkdir -p /home/sandbox/.ssh
    cp /root/.ssh/authorized_keys /home/sandbox/.ssh/
    chown -R sandbox:sandbox /home/sandbox/.ssh
    chmod 700 /home/sandbox/.ssh
    chmod 600 /home/sandbox/.ssh/authorized_keys

    # Create uploads directory
    mkdir -p /home/sandbox/uploads
    chown sandbox:sandbox /home/sandbox/uploads

    # Enable SSH
    systemctl enable ssh
'
```

### 4.5 Security Hardening

```bash
pct exec 9000 -- bash -c '
    # Process limits (prevent fork bombs)
    echo "* soft nproc 256" >> /etc/security/limits.conf
    echo "* hard nproc 512" >> /etc/security/limits.conf

    # Disable core dumps
    echo "* hard core 0" >> /etc/security/limits.conf

    # Disable unnecessary services
    systemctl disable systemd-resolved 2>/dev/null || true
    systemctl disable snapd 2>/dev/null || true
'
```

### 4.6 Convert to Template

```bash
pct stop 9000
pct template 9000
```

### 4.7 Verify Template

Clone and test manually:
```bash
pct clone 9000 9999 --hostname test-sandbox --full
pct start 9999
# Wait for DHCP, then SSH in
ssh sandbox@<container-ip>
# Run some commands, verify packages installed
sudo apt-get update
python3 --version
node --version
# Clean up
exit
pct stop 9999
pct destroy 9999
```

---

## 5. SSH Key Setup

### 5.1 Generate Key Pair

```bash
ssh-keygen -t ed25519 -f /etc/mort/sandbox_key -N "" -C "mort-sandbox"
```

### 5.2 Install Public Key in Template

This was done in step 4.2 with `--ssh-public-keys`. If you need to update it:

```bash
pct start 9000  # Only if template — you'll need to untemplate first
# Copy key
cat /etc/mort/sandbox_key.pub | pct exec 9000 -- tee /home/sandbox/.ssh/authorized_keys
pct exec 9000 -- chown sandbox:sandbox /home/sandbox/.ssh/authorized_keys
pct exec 9000 -- chmod 600 /home/sandbox/.ssh/authorized_keys
pct stop 9000
pct template 9000
```

### 5.3 Set Permissions

```bash
chmod 600 /etc/mort/sandbox_key
chmod 644 /etc/mort/sandbox_key.pub
# If running as a specific user:
chown mort:mort /etc/mort/sandbox_key /etc/mort/sandbox_key.pub
```

---

## 6. Configuration

### Go Configuration

```go
signer, _ := sandbox.LoadSSHKey("/etc/mort/sandbox_key")

mgr, _ := sandbox.NewManager(sandbox.Config{
    Proxmox: sandbox.ProxmoxConfig{
        BaseURL:            "https://proxmox.local:8006",
        TokenID:            "mort-sandbox@pve!sandbox-token",
        Secret:             os.Getenv("SANDBOX_PROXMOX_SECRET"),
        Node:               "pve",
        TemplateID:         9000,
        Pool:               "sandbox-pool",
        Bridge:             "vmbr1",
        InsecureSkipVerify: true, // Only for self-signed certs
    },
    SSH: sandbox.SSHConfig{
        Signer:         signer,
        User:           "sandbox",  // default
        ConnectTimeout: 10 * time.Second,  // default
        CommandTimeout: 60 * time.Second,  // default
    },
    Defaults: sandbox.ContainerConfig{
        CPUs:     1,
        MemoryMB: 1024,
        DiskGB:   8,
    },
})
```

### Environment Variables

| Variable | Description |
|----------|-------------|
| `SANDBOX_PROXMOX_SECRET` | Proxmox API token secret |
| `SANDBOX_SSH_KEY_PATH` | Path to SSH private key (alternative to config) |

---

## 7. Hardening Checklist

Run through this checklist after setup:

### Container Isolation
- [ ] Containers are unprivileged (verify UID mapping in `/etc/pve/lxc/<id>.conf`)
- [ ] Nesting is disabled (`features: nesting=0`)
- [ ] Swap is disabled on containers (`swap: 0`)
- [ ] Resource pool scoping: API token can only touch `sandbox-pool`

### Network Isolation
- [ ] `vmbr1` has no physical ports (`bridge-ports none`)
- [ ] nftables rules loaded: `nft list ruleset` shows sandbox table
- [ ] nftables persists across reboots: `systemctl is-enabled nftables`
- [ ] Default-deny outbound for containers
- [ ] DNS (port 53) allowed for all containers
- [ ] HTTP/HTTPS only for containers in `internet_allowed` set
- [ ] Rate limiting active (50 conn/sec)
- [ ] Containers cannot reach Proxmox management (port 8006 blocked)

### Security Profiles
- [ ] AppArmor profile active: `lxc-container-default-cgns`
- [ ] Process limits in `/etc/security/limits.conf` (nproc 256/512)
- [ ] Core dumps disabled
- [ ] Capability drops verified in container config

### Functional Tests
- [ ] **Fork bomb test**: run `:(){ :|:& };:` in container → PID limit fires, container survives
- [ ] **OOM test**: allocate >1GB memory → container OOM-killed, host unaffected
- [ ] **Network scan test**: `nmap` from container → blocked by nftables
- [ ] **Container escape test**: attempt to mount host filesystem → denied
- [ ] **LAN access test**: ping LAN hosts → blocked
- [ ] **Cross-container test**: ping other sandbox containers → blocked
- [ ] **Internet access test**: HTTP without being in `internet_allowed` → blocked
- [ ] **Internet access test**: add to `internet_allowed` → HTTP works
- [ ] **Cleanup test**: destroy container → verify no orphan volumes

### Operational Tests
- [ ] Clone template → container starts → SSH connects → commands work
- [ ] File upload/download via SFTP works
- [ ] Container destroy removes all resources
- [ ] Orphan cleanup: kill application mid-session, restart, verify cleanup

---

## 8. Monitoring & Maintenance

### Log Rotation

Sandbox session logs should be rotated to prevent disk exhaustion. If using slog to a file:

```
# /etc/logrotate.d/sandbox
/var/log/sandbox/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
}
```

### Storage Cleanup

Verify destroyed containers don't leave orphan volumes:
```bash
# List all LVM volumes in the sandbox storage
lvs | grep sandbox

# Compare with running containers
pct list | grep sandbox
```

### Template Updates

Periodically update the template with latest packages:

```bash
# Un-template (creates a regular container from template)
# Note: you can't un-template directly; clone then replace
pct clone 9000 9001 --hostname template-update --full
pct start 9001
pct exec 9001 -- bash -c 'apt-get update && apt-get upgrade -y && apt-get clean'
pct stop 9001

# Destroy old template and create new one
pct destroy 9000
# Rename 9001 → 9000 (or update your config to use the new ID)
pct template 9001
```

### Proxmox Host Updates

```bash
apt-get update && apt-get dist-upgrade -y
# Reboot if kernel was updated
# Verify nftables rules are still loaded after reboot
nft list ruleset
```

---

## 9. Troubleshooting

### Container won't start

```bash
# Check task log
pct start <id>
# If error, check:
journalctl -u pve-container@<id> -n 50

# Common issues:
# - Storage full: check `df -h` and `lvs`
# - UID mapping issues: verify /etc/subuid and /etc/subgid
```

### SSH connection refused

```bash
# Verify container is running
pct status <id>

# Check if SSH is running inside container
pct exec <id> -- systemctl status ssh

# Verify IP assignment
pct exec <id> -- ip addr show eth0

# Check DHCP leases
cat /var/lib/misc/dnsmasq.leases
```

### Container has no internet (when it should)

```bash
# Verify container IP is in the internet_allowed set
nft list set inet sandbox internet_allowed

# Manually add for testing
nft add element inet sandbox internet_allowed { 10.99.1.5 }

# Verify NAT is working
nft list table nat

# Check if IP forwarding is enabled
cat /proc/sys/net/ipv4/ip_forward  # Should be 1
```

### nftables rules lost after reboot

```bash
# Verify nftables is enabled
systemctl is-enabled nftables

# If rules are missing, reload
nft -f /etc/nftables.conf

# Make sure the config file is correct
nft -c -f /etc/nftables.conf  # Check syntax without applying
```

### Orphaned containers

```bash
# List all containers in the sandbox pool
pvesh get /pools/sandbox-pool --output-format json | jq '.members[] | select(.type == "lxc")'

# Destroy orphans manually
pct stop <id> && pct destroy <id> --force --purge
```