Add sandbox package for isolated Linux containers via Proxmox LXC
Provides a complete lifecycle manager for ephemeral sandbox environments: - ProxmoxClient: thin REST wrapper for container CRUD, IP discovery, internet toggle - SSHExecutor: persistent SSH/SFTP for command execution and file transfer - Manager/Sandbox: high-level orchestrator tying Proxmox + SSH together - 22 unit tests with mock Proxmox HTTP server - Proxmox setup & hardening guide (docs/sandbox-setup.md) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
575
docs/sandbox-setup.md
Normal file
575
docs/sandbox-setup.md
Normal file
@@ -0,0 +1,575 @@
|
||||
# Sandbox Setup & Hardening Guide
|
||||
|
||||
Complete guide for setting up a Proxmox VE host to run isolated LXC sandbox containers for the go-llm sandbox package.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Prerequisites](#1-prerequisites)
|
||||
2. [Proxmox Host Preparation](#2-proxmox-host-preparation)
|
||||
3. [Network Setup](#3-network-setup)
|
||||
4. [LXC Template Creation](#4-lxc-template-creation)
|
||||
5. [SSH Key Setup](#5-ssh-key-setup)
|
||||
6. [Configuration](#6-configuration)
|
||||
7. [Hardening Checklist](#7-hardening-checklist)
|
||||
8. [Monitoring & Maintenance](#8-monitoring--maintenance)
|
||||
9. [Troubleshooting](#9-troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
### Hardware/VM Requirements
|
||||
|
||||
| Resource | Minimum | Recommended |
|
||||
|----------|---------|-------------|
|
||||
| CPU | 4 cores | 8+ cores |
|
||||
| RAM | 8 GB | 16+ GB |
|
||||
| Storage | 100 GB SSD | 250+ GB SSD |
|
||||
| Network | 1 NIC | 2 NICs (mgmt + sandbox) |
|
||||
|
||||
### Software
|
||||
|
||||
- Proxmox VE 8.x ([installation guide](https://pve.proxmox.com/wiki/Installation))
|
||||
- During install, configure the management interface on `vmbr0`
|
||||
|
||||
---
|
||||
|
||||
## 2. Proxmox Host Preparation
|
||||
|
||||
### Create Resource Pool
|
||||
|
||||
Scope sandbox containers to a dedicated resource pool to limit API token access:
|
||||
|
||||
```bash
|
||||
pvesh create /pools --poolid sandbox-pool
|
||||
```
|
||||
|
||||
### Create API User and Token
|
||||
|
||||
```bash
|
||||
# Create dedicated user
|
||||
pveum useradd mort-sandbox@pve
|
||||
|
||||
# Create role with minimum required permissions
|
||||
pveum roleadd SandboxAdmin -privs "VM.Allocate,VM.Clone,VM.Audit,VM.PowerMgmt,VM.Console,Datastore.AllocateSpace,Datastore.Audit"
|
||||
|
||||
# Grant role on the sandbox pool only
|
||||
pveum aclmod /pool/sandbox-pool -user mort-sandbox@pve -role SandboxAdmin
|
||||
|
||||
# Grant access to the template storage
|
||||
pveum aclmod /storage/local -user mort-sandbox@pve -role PVEDatastoreUser
|
||||
|
||||
# Create API token (privsep=0 means token inherits user's permissions)
|
||||
pveum user token add mort-sandbox@pve sandbox-token --privsep=0
|
||||
```
|
||||
|
||||
Save the output — it contains the token secret:
|
||||
```
|
||||
┌──────────┬──────────────────────────────────────────┐
|
||||
│ key │ value │
|
||||
╞══════════╪══════════════════════════════════════════╡
|
||||
│ full-tokenid │ mort-sandbox@pve!sandbox-token │
|
||||
│ value │ xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx │
|
||||
└──────────┴──────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Store the secret securely (environment variable, secret manager, etc.). Never commit it to source control.
|
||||
|
||||
---
|
||||
|
||||
## 3. Network Setup
|
||||
|
||||
### 3.1 Create Isolated Bridge
|
||||
|
||||
Add to `/etc/network/interfaces` on the Proxmox host:
|
||||
|
||||
```
|
||||
auto vmbr1
|
||||
iface vmbr1 inet static
|
||||
address 10.99.0.1/16
|
||||
bridge-ports none
|
||||
bridge-stp off
|
||||
bridge-fd 0
|
||||
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
|
||||
# NAT for optional internet access (controlled per-container by nftables)
|
||||
post-up nft add table nat 2>/dev/null; true
|
||||
post-up nft add chain nat postrouting { type nat hook postrouting priority 100 \; } 2>/dev/null; true
|
||||
post-up nft add rule nat postrouting oifname "vmbr0" ip saddr 10.99.0.0/16 masquerade 2>/dev/null; true
|
||||
```
|
||||
|
||||
Apply the configuration:
|
||||
```bash
|
||||
ifreload -a
|
||||
```
|
||||
|
||||
### 3.2 Install and Configure DHCP
|
||||
|
||||
```bash
|
||||
apt-get install -y dnsmasq
|
||||
```
|
||||
|
||||
Create `/etc/dnsmasq.d/sandbox.conf`:
|
||||
```
|
||||
interface=vmbr1
|
||||
bind-interfaces
|
||||
dhcp-range=10.99.1.1,10.99.254.254,255.255.0.0,1h
|
||||
dhcp-option=option:router,10.99.0.1
|
||||
dhcp-option=option:dns-server,1.1.1.1,8.8.8.8
|
||||
```
|
||||
|
||||
Restart dnsmasq:
|
||||
```bash
|
||||
systemctl restart dnsmasq
|
||||
systemctl enable dnsmasq
|
||||
```
|
||||
|
||||
### 3.3 Configure nftables Firewall
|
||||
|
||||
Create `/etc/nftables.conf`:
|
||||
|
||||
```nft
|
||||
#!/usr/sbin/nft -f
|
||||
|
||||
flush ruleset
|
||||
|
||||
table inet sandbox {
|
||||
# Dynamic set of container IPs allowed internet access.
|
||||
# Populated/cleared by the sandbox manager via the Proxmox API.
|
||||
set internet_allowed {
|
||||
type ipv4_addr
|
||||
}
|
||||
|
||||
chain forward {
|
||||
type filter hook forward priority 0; policy drop;
|
||||
|
||||
# Allow established/related connections
|
||||
ct state established,related accept
|
||||
|
||||
# Allow inter-bridge traffic (host ↔ containers via vmbr1)
|
||||
iifname "vmbr1" oifname "vmbr1" accept
|
||||
|
||||
# Allow DNS for all containers (needed for apt)
|
||||
ip saddr 10.99.0.0/16 udp dport 53 accept
|
||||
ip saddr 10.99.0.0/16 tcp dport 53 accept
|
||||
|
||||
# Allow HTTP/HTTPS only for containers in the internet_allowed set
|
||||
ip saddr @internet_allowed tcp dport { 80, 443 } accept
|
||||
|
||||
# Rate limit: max 50 new connections per second per container
|
||||
ip saddr 10.99.0.0/16 ct state new limit rate over 50/second drop
|
||||
|
||||
# Block everything else from containers
|
||||
ip saddr 10.99.0.0/16 drop
|
||||
|
||||
# Allow host → containers (for SSH from the application)
|
||||
ip daddr 10.99.0.0/16 accept
|
||||
}
|
||||
|
||||
chain input {
|
||||
type filter hook input priority 0; policy accept;
|
||||
|
||||
# Block containers from accessing Proxmox management ports
|
||||
# (only SSH is allowed for the sandbox manager)
|
||||
iifname "vmbr1" ip daddr 10.99.0.1 tcp dport != 22 drop
|
||||
}
|
||||
}
|
||||
|
||||
# NAT table for optional internet access
|
||||
table nat {
|
||||
chain postrouting {
|
||||
type nat hook postrouting priority 100;
|
||||
oifname "vmbr0" ip saddr 10.99.0.0/16 masquerade
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Apply and persist:
|
||||
```bash
|
||||
nft -f /etc/nftables.conf
|
||||
systemctl enable nftables
|
||||
```
|
||||
|
||||
Verify:
|
||||
```bash
|
||||
nft list ruleset
|
||||
```
|
||||
|
||||
### 3.4 Test Network Isolation
|
||||
|
||||
From a test container on `vmbr1`:
|
||||
```bash
|
||||
# Should work: DNS resolution
|
||||
dig google.com
|
||||
|
||||
# Should be blocked: HTTP (not in internet_allowed set)
|
||||
curl -s --connect-timeout 5 https://google.com && echo "FAIL: should be blocked" || echo "OK: blocked"
|
||||
|
||||
# Should be blocked: access to LAN
|
||||
ping -c 1 -W 2 192.168.1.1 && echo "FAIL: LAN reachable" || echo "OK: LAN blocked"
|
||||
|
||||
# Should be blocked: access to Proxmox management
|
||||
curl -s --connect-timeout 5 https://10.99.0.1:8006 && echo "FAIL: Proxmox reachable" || echo "OK: Proxmox blocked"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. LXC Template Creation
|
||||
|
||||
### 4.1 Download Base Image
|
||||
|
||||
```bash
|
||||
pveam update
|
||||
pveam download local ubuntu-24.04-standard_24.04-1_amd64.tar.zst
|
||||
```
|
||||
|
||||
### 4.2 Create Template Container
|
||||
|
||||
```bash
|
||||
pct create 9000 local:vztmpl/ubuntu-24.04-standard_24.04-1_amd64.tar.zst \
|
||||
--hostname sandbox-template \
|
||||
--memory 1024 \
|
||||
--swap 0 \
|
||||
--cores 1 \
|
||||
--rootfs local-lvm:8 \
|
||||
--net0 name=eth0,bridge=vmbr1,ip=dhcp \
|
||||
--unprivileged 1 \
|
||||
--features nesting=0 \
|
||||
--ostype ubuntu \
|
||||
--ssh-public-keys /root/.ssh/mort_sandbox.pub \
|
||||
--pool sandbox-pool \
|
||||
--start 0
|
||||
```
|
||||
|
||||
### 4.3 Install Base Packages
|
||||
|
||||
```bash
|
||||
pct start 9000
|
||||
pct exec 9000 -- bash -c '
|
||||
apt-get update && apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
python3 python3-pip python3-venv \
|
||||
nodejs npm \
|
||||
git curl wget jq \
|
||||
vim nano \
|
||||
htop tree \
|
||||
ca-certificates \
|
||||
openssh-server \
|
||||
&& apt-get clean \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
'
|
||||
```
|
||||
|
||||
### 4.4 Create Sandbox User
|
||||
|
||||
```bash
|
||||
pct exec 9000 -- bash -c '
|
||||
# Create unprivileged sandbox user with sudo
|
||||
useradd -m -s /bin/bash -G sudo sandbox
|
||||
echo "sandbox ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/sandbox
|
||||
|
||||
# Set up SSH access
|
||||
mkdir -p /home/sandbox/.ssh
|
||||
cp /root/.ssh/authorized_keys /home/sandbox/.ssh/
|
||||
chown -R sandbox:sandbox /home/sandbox/.ssh
|
||||
chmod 700 /home/sandbox/.ssh
|
||||
chmod 600 /home/sandbox/.ssh/authorized_keys
|
||||
|
||||
# Create uploads directory
|
||||
mkdir -p /home/sandbox/uploads
|
||||
chown sandbox:sandbox /home/sandbox/uploads
|
||||
|
||||
# Enable SSH
|
||||
systemctl enable ssh
|
||||
'
|
||||
```
|
||||
|
||||
### 4.5 Security Hardening
|
||||
|
||||
```bash
|
||||
pct exec 9000 -- bash -c '
|
||||
# Process limits (prevent fork bombs)
|
||||
echo "* soft nproc 256" >> /etc/security/limits.conf
|
||||
echo "* hard nproc 512" >> /etc/security/limits.conf
|
||||
|
||||
# Disable core dumps
|
||||
echo "* hard core 0" >> /etc/security/limits.conf
|
||||
|
||||
# Disable unnecessary services
|
||||
systemctl disable systemd-resolved 2>/dev/null || true
|
||||
systemctl disable snapd 2>/dev/null || true
|
||||
'
|
||||
```
|
||||
|
||||
### 4.6 Convert to Template
|
||||
|
||||
```bash
|
||||
pct stop 9000
|
||||
pct template 9000
|
||||
```
|
||||
|
||||
### 4.7 Verify Template
|
||||
|
||||
Clone and test manually:
|
||||
```bash
|
||||
pct clone 9000 9999 --hostname test-sandbox --full
|
||||
pct start 9999
|
||||
# Wait for DHCP, then SSH in
|
||||
ssh sandbox@<container-ip>
|
||||
# Run some commands, verify packages installed
|
||||
sudo apt-get update
|
||||
python3 --version
|
||||
node --version
|
||||
# Clean up
|
||||
exit
|
||||
pct stop 9999
|
||||
pct destroy 9999
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. SSH Key Setup
|
||||
|
||||
### 5.1 Generate Key Pair
|
||||
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -f /etc/mort/sandbox_key -N "" -C "mort-sandbox"
|
||||
```
|
||||
|
||||
### 5.2 Install Public Key in Template
|
||||
|
||||
This was done in step 4.2 with `--ssh-public-keys`. If you need to update it:
|
||||
|
||||
```bash
|
||||
pct start 9000 # Only if template — you'll need to untemplate first
|
||||
# Copy key
|
||||
cat /etc/mort/sandbox_key.pub | pct exec 9000 -- tee /home/sandbox/.ssh/authorized_keys
|
||||
pct exec 9000 -- chown sandbox:sandbox /home/sandbox/.ssh/authorized_keys
|
||||
pct exec 9000 -- chmod 600 /home/sandbox/.ssh/authorized_keys
|
||||
pct stop 9000
|
||||
pct template 9000
|
||||
```
|
||||
|
||||
### 5.3 Set Permissions
|
||||
|
||||
```bash
|
||||
chmod 600 /etc/mort/sandbox_key
|
||||
chmod 644 /etc/mort/sandbox_key.pub
|
||||
# If running as a specific user:
|
||||
chown mort:mort /etc/mort/sandbox_key /etc/mort/sandbox_key.pub
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Configuration
|
||||
|
||||
### Go Configuration
|
||||
|
||||
```go
|
||||
signer, _ := sandbox.LoadSSHKey("/etc/mort/sandbox_key")
|
||||
|
||||
mgr, _ := sandbox.NewManager(sandbox.Config{
|
||||
Proxmox: sandbox.ProxmoxConfig{
|
||||
BaseURL: "https://proxmox.local:8006",
|
||||
TokenID: "mort-sandbox@pve!sandbox-token",
|
||||
Secret: os.Getenv("SANDBOX_PROXMOX_SECRET"),
|
||||
Node: "pve",
|
||||
TemplateID: 9000,
|
||||
Pool: "sandbox-pool",
|
||||
Bridge: "vmbr1",
|
||||
InsecureSkipVerify: true, // Only for self-signed certs
|
||||
},
|
||||
SSH: sandbox.SSHConfig{
|
||||
Signer: signer,
|
||||
User: "sandbox", // default
|
||||
ConnectTimeout: 10 * time.Second, // default
|
||||
CommandTimeout: 60 * time.Second, // default
|
||||
},
|
||||
Defaults: sandbox.ContainerConfig{
|
||||
CPUs: 1,
|
||||
MemoryMB: 1024,
|
||||
DiskGB: 8,
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `SANDBOX_PROXMOX_SECRET` | Proxmox API token secret |
|
||||
| `SANDBOX_SSH_KEY_PATH` | Path to SSH private key (alternative to config) |
|
||||
|
||||
---
|
||||
|
||||
## 7. Hardening Checklist
|
||||
|
||||
Run through this checklist after setup:
|
||||
|
||||
### Container Isolation
|
||||
- [ ] Containers are unprivileged (verify UID mapping in `/etc/pve/lxc/<id>.conf`)
|
||||
- [ ] Nesting is disabled (`features: nesting=0`)
|
||||
- [ ] Swap is disabled on containers (`swap: 0`)
|
||||
- [ ] Resource pool scoping: API token can only touch `sandbox-pool`
|
||||
|
||||
### Network Isolation
|
||||
- [ ] `vmbr1` has no physical ports (`bridge-ports none`)
|
||||
- [ ] nftables rules loaded: `nft list ruleset` shows sandbox table
|
||||
- [ ] nftables persists across reboots: `systemctl is-enabled nftables`
|
||||
- [ ] Default-deny outbound for containers
|
||||
- [ ] DNS (port 53) allowed for all containers
|
||||
- [ ] HTTP/HTTPS only for containers in `internet_allowed` set
|
||||
- [ ] Rate limiting active (50 conn/sec)
|
||||
- [ ] Containers cannot reach Proxmox management (port 8006 blocked)
|
||||
|
||||
### Security Profiles
|
||||
- [ ] AppArmor profile active: `lxc-container-default-cgns`
|
||||
- [ ] Process limits in `/etc/security/limits.conf` (nproc 256/512)
|
||||
- [ ] Core dumps disabled
|
||||
- [ ] Capability drops verified in container config
|
||||
|
||||
### Functional Tests
|
||||
- [ ] **Fork bomb test**: run `:(){ :|:& };:` in container → PID limit fires, container survives
|
||||
- [ ] **OOM test**: allocate >1GB memory → container OOM-killed, host unaffected
|
||||
- [ ] **Network scan test**: `nmap` from container → blocked by nftables
|
||||
- [ ] **Container escape test**: attempt to mount host filesystem → denied
|
||||
- [ ] **LAN access test**: ping LAN hosts → blocked
|
||||
- [ ] **Cross-container test**: ping other sandbox containers → blocked
|
||||
- [ ] **Internet access test**: HTTP without being in `internet_allowed` → blocked
|
||||
- [ ] **Internet access test**: add to `internet_allowed` → HTTP works
|
||||
- [ ] **Cleanup test**: destroy container → verify no orphan volumes
|
||||
|
||||
### Operational Tests
|
||||
- [ ] Clone template → container starts → SSH connects → commands work
|
||||
- [ ] File upload/download via SFTP works
|
||||
- [ ] Container destroy removes all resources
|
||||
- [ ] Orphan cleanup: kill application mid-session, restart, verify cleanup
|
||||
|
||||
---
|
||||
|
||||
## 8. Monitoring & Maintenance
|
||||
|
||||
### Log Rotation
|
||||
|
||||
Sandbox session logs should be rotated to prevent disk exhaustion. If using slog to a file:
|
||||
|
||||
```
|
||||
# /etc/logrotate.d/sandbox
|
||||
/var/log/sandbox/*.log {
|
||||
daily
|
||||
rotate 14
|
||||
compress
|
||||
delaycompress
|
||||
missingok
|
||||
notifempty
|
||||
}
|
||||
```
|
||||
|
||||
### Storage Cleanup
|
||||
|
||||
Verify destroyed containers don't leave orphan volumes:
|
||||
```bash
|
||||
# List all LVM volumes in the sandbox storage
|
||||
lvs | grep sandbox
|
||||
|
||||
# Compare with running containers
|
||||
pct list | grep sandbox
|
||||
```
|
||||
|
||||
### Template Updates
|
||||
|
||||
Periodically update the template with latest packages:
|
||||
|
||||
```bash
|
||||
# Un-template (creates a regular container from template)
|
||||
# Note: you can't un-template directly; clone then replace
|
||||
pct clone 9000 9001 --hostname template-update --full
|
||||
pct start 9001
|
||||
pct exec 9001 -- bash -c 'apt-get update && apt-get upgrade -y && apt-get clean'
|
||||
pct stop 9001
|
||||
|
||||
# Destroy old template and create new one
|
||||
pct destroy 9000
|
||||
# Rename 9001 → 9000 (or update your config to use the new ID)
|
||||
pct template 9001
|
||||
```
|
||||
|
||||
### Proxmox Host Updates
|
||||
|
||||
```bash
|
||||
apt-get update && apt-get dist-upgrade -y
|
||||
# Reboot if kernel was updated
|
||||
# Verify nftables rules are still loaded after reboot
|
||||
nft list ruleset
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Troubleshooting
|
||||
|
||||
### Container won't start
|
||||
|
||||
```bash
|
||||
# Check task log
|
||||
pct start <id>
|
||||
# If error, check:
|
||||
journalctl -u pve-container@<id> -n 50
|
||||
|
||||
# Common issues:
|
||||
# - Storage full: check `df -h` and `lvs`
|
||||
# - UID mapping issues: verify /etc/subuid and /etc/subgid
|
||||
```
|
||||
|
||||
### SSH connection refused
|
||||
|
||||
```bash
|
||||
# Verify container is running
|
||||
pct status <id>
|
||||
|
||||
# Check if SSH is running inside container
|
||||
pct exec <id> -- systemctl status ssh
|
||||
|
||||
# Verify IP assignment
|
||||
pct exec <id> -- ip addr show eth0
|
||||
|
||||
# Check DHCP leases
|
||||
cat /var/lib/misc/dnsmasq.leases
|
||||
```
|
||||
|
||||
### Container has no internet (when it should)
|
||||
|
||||
```bash
|
||||
# Verify container IP is in the internet_allowed set
|
||||
nft list set inet sandbox internet_allowed
|
||||
|
||||
# Manually add for testing
|
||||
nft add element inet sandbox internet_allowed { 10.99.1.5 }
|
||||
|
||||
# Verify NAT is working
|
||||
nft list table nat
|
||||
|
||||
# Check if IP forwarding is enabled
|
||||
cat /proc/sys/net/ipv4/ip_forward # Should be 1
|
||||
```
|
||||
|
||||
### nftables rules lost after reboot
|
||||
|
||||
```bash
|
||||
# Verify nftables is enabled
|
||||
systemctl is-enabled nftables
|
||||
|
||||
# If rules are missing, reload
|
||||
nft -f /etc/nftables.conf
|
||||
|
||||
# Make sure the config file is correct
|
||||
nft -c -f /etc/nftables.conf # Check syntax without applying
|
||||
```
|
||||
|
||||
### Orphaned containers
|
||||
|
||||
```bash
|
||||
# List all containers in the sandbox pool
|
||||
pvesh get /pools/sandbox-pool --output-format json | jq '.members[] | select(.type == "lxc")'
|
||||
|
||||
# Destroy orphans manually
|
||||
pct stop <id> && pct destroy <id> --force --purge
|
||||
```
|
||||
Reference in New Issue
Block a user