# Container Bootstrap Plan — Automating GB Container Deployment ## Overview This document describes how to improve `installer.rs` to automate the deployment of General Bots containers on Incus. The goal is to replicate what was done manually during the pragmatismo migration from LXD to Incus. --- # rust incus install base sudo apt install -y curl gnupg sudo mkdir -p /etc/apt/keyrings sudo curl -fsSL https://pkgs.zabbly.com/key.asc -o /etc/apt/keyrings/zabbly.asc sudo sh -c 'cat > /etc/apt/sources.list.d/zabbly-incus-stable.sources << EOF Enabled: yes Types: deb URIs: https://pkgs.zabbly.com/incus/stable Suites: bookworm Components: main Architectures: $(dpkg --print-architecture) Signed-By: /etc/apt/keyrings/zabbly.asc EOF' sudo apt update sudo apt install incus ## What Was Done Manually (Reference Implementation) ### Migration Summary (pragmatismo tenant) | Item | Detail | |------|--------| | Source | LXD 5.21 @ 82.29.59.188 | | Destination | Incus 6.x @ 63.141.255.9 | | Method | `incus copy --instance-only lxd-source:` | | Data transfer | tar.gz → push to containers | | Containers | 10 (dns, email, webmail, alm, drive, tables, system, proxy, alm-ci, table-editor) | | Total data | ~44 GB | --- ## Container Architecture (Reference) ### Container Types & Services | Container | Purpose | Ports | Service Binary | Service User | |-----------|---------|-------|---------------|-------------| | **dns** | CoreDNS | 53 | `/opt/gbo/bin/coredns` | root | | **email** | Stalwart mail | 25,143,465,587,993,995,110,4190 | `/opt/gbo/bin/stalwart` | root | | **webmail** | Roundcube/PHP | 80,443 | Apache (`/usr/sbin/apache2`) | www-data | | **alm** | Forgejo ALM | 4747 | `/opt/gbo/bin/forgejo` | gbuser | | **drive** | MinIO S3 | 9000,9001 | `/opt/gbo/bin/minio` | root | | **tables** | PostgreSQL | 5432 | system-installed | root | | **system** | botserver + stack | 5858, 8200, 6379, 6333, 9100 | `/opt/gbo/bin/botserver` | gbuser | | **proxy** | Caddy | 80, 443 | `/usr/bin/caddy` | gbuser | | **alm-ci** | Forgejo runner | none | `/opt/gbo/bin/forgejo-runner` | root | | **table-editor** | NocoDB | 8080 | system-installed | root | **RULE: ALL services run as gbuser where possible, ALL data under /opt/gbo, Service name = container name (e.g., proxy-caddy.service)** ### Network Layout ``` Host (63.141.255.9) ├── Incus bridge (10.107.115.x) │ ├── dns (10.107.115.155) │ ├── email (10.107.115.200) │ ├── webmail (10.107.115.87) │ ├── alm (10.107.115.4) │ ├── drive (10.107.115.114) │ ├── tables (10.107.115.33) │ ├── system (10.107.115.229) │ ├── proxy (10.107.115.189) │ ├── alm-ci (10.107.115.190) │ └── table-editor (10.107.115.73) └── iptables NAT → external ports ``` --- ## Key Paths (Must Match Production) Inside each container: ``` /opt/gbo/ ├── bin/ # binaries (coredns, stalwart, forgejo, caddy, minio, postgres) ├── conf/ # service configs (Corefile, config.toml, app.ini) ├── data/ # app data (zone files, databases, repos) └── logs/ # service logs ``` On host: ``` /opt/gbo/tenants// ├── dns/ │ ├── bin/ │ ├── conf/ │ ├── data/ │ └── logs/ ├── email/ ├── webmail/ ├── alm/ ├── drive/ ├── tables/ ├── system/ ├── proxy/ ├── alm-ci/ └── table-editor/ ``` --- ## Service Files (Templates) **RULE: ALL services run as gbuser where possible, Service name = container name (e.g., dns.service, proxy-caddy.service)** ### dns.service (CoreDNS) ```ini [Unit] Description=CoreDNS After=network.target [Service] User=root WorkingDirectory=/opt/gbo ExecStart=/opt/gbo/bin/coredns -conf /opt/gbo/conf/Corefile Restart=always RestartSec=5 [Install] WantedBy=multi-user.target ``` ### email.service (Stalwart) ```ini [Unit] Description=Stalwart Mail Server After=network.target [Service] Type=simple User=root WorkingDirectory=/opt/gbo ExecStart=/opt/gbo/bin/stalwart --config /opt/gbo/conf/config.toml Restart=always RestartSec=5 [Install] WantedBy=multi-user.target ``` ### proxy-caddy.service ```ini [Unit] Description=Caddy Reverse Proxy After=network.target [Service] User=gbuser Group=gbuser WorkingDirectory=/opt/gbo ExecStart=/usr/bin/caddy run --config /opt/gbo/conf/config --adapter caddyfile Restart=always RestartSec=5 [Install] WantedBy=multi-user.target ``` ### alm.service (Forgejo) ```ini [Unit] Description=Forgejo Git Server After=network.target [Service] User=gbuser Group=gbuser WorkingDirectory=/opt/gbo ExecStart=/opt/gbo/bin/forgejo web --config /opt/gbo/conf/app.ini Restart=always RestartSec=5 [Install] WantedBy=multi-user.target ``` ### drive-minio.service ```ini [Unit] Description=MinIO Object Storage After=network-online.target Wants=network-online.target [Service] User=gbuser Group=gbuser WorkingDirectory=/opt/gbo ExecStart=/opt/gbo/bin/minio server --console-address :4646 /opt/gbo/data Restart=always RestartSec=10 [Install] WantedBy=multi-user.target ``` ### tables-postgresql.service ```ini [Unit] Description=PostgreSQL After=network.target [Service] User=gbuser Group=gbuser WorkingDirectory=/opt/gbo ExecStart=/opt/gbo/bin/postgres -D /opt/gbo/data -c config_file=/opt/gbo/conf/postgresql.conf Restart=always RestartSec=5 [Install] WantedBy=multi-user.target ``` ### webmail-apache.service ```ini [Unit] Description=Apache Webmail After=network.target [Service] User=www-data Group=www-data WorkingDirectory=/var/www/html ExecStart=/usr/sbin/apache2 -D FOREGROUND Restart=always RestartSec=5 [Install] WantedBy=multi-user.target ``` --- ## iptables NAT Rules (CRITICAL - Use ONLY iptables, NEVER socat or Incus proxy devices) ### Prerequisites ```bash # Enable IP forwarding (persistent) echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.d/99-ipforward.conf sudo sysctl -w net.ipv4.ip_forward=1 # Enable route_localnet for NAT to work with localhost echo "net.ipv4.conf.all.route_localnet = 1" | sudo tee /etc/sysctl.d/99-localnet.conf sudo sysctl -w net.ipv4.conf.all.route_localnet=1 ``` ### Required NAT Rules (Complete Set) ```bash # ================== # DNS (dns container) # ================== sudo iptables -t nat -A PREROUTING -p udp --dport 53 -j DNAT --to-destination 10.107.115.155:53 sudo iptables -t nat -A PREROUTING -p tcp --dport 53 -j DNAT --to-destination 10.107.115.155:53 sudo iptables -t nat -A OUTPUT -p udp --dport 53 -j DNAT --to-destination 10.107.115.155:53 sudo iptables -t nat -A OUTPUT -p tcp --dport 53 -j DNAT --to-destination 10.107.115.155:53 # ================== # Tables (PostgreSQL) - External port 4445 # ================== sudo iptables -t nat -A PREROUTING -p tcp --dport 4445 -j DNAT --to-destination 10.107.115.33:5432 sudo iptables -t nat -A OUTPUT -p tcp --dport 4445 -j DNAT --to-destination 10.107.115.33:5432 # ================== # Proxy (Caddy) - 80, 443 # ================== sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 10.107.115.189:80 sudo iptables -t nat -A PREROUTING -p tcp --dport 443 -j DNAT --to-destination 10.107.115.189:443 # ================== # Email (email container) - Stalwart # ================== sudo iptables -t nat -A PREROUTING -p tcp --dport 25 -j DNAT --to-destination 10.107.115.200:25 sudo iptables -t nat -A PREROUTING -p tcp --dport 465 -j DNAT --to-destination 10.107.115.200:465 sudo iptables -t nat -A PREROUTING -p tcp --dport 587 -j DNAT --to-destination 10.107.115.200:587 sudo iptables -t nat -A PREROUTING -p tcp --dport 993 -j DNAT --to-destination 10.107.115.200:993 sudo iptables -t nat -A PREROUTING -p tcp --dport 995 -j DNAT --to-destination 10.107.115.200:995 sudo iptables -t nat -A PREROUTING -p tcp --dport 143 -j DNAT --to-destination 10.107.115.200:143 sudo iptables -t nat -A PREROUTING -p tcp --dport 110 -j DNAT --to-destination 10.107.115.200:110 sudo iptables -t nat -A PREROUTING -p tcp --dport 4190 -j DNAT --to-destination 10.107.115.200:4190 # ================== # FORWARD rules (required for containers to receive traffic) # ================== sudo iptables -A FORWARD -p tcp -d 10.107.115.155 --dport 53 -j ACCEPT sudo iptables -A FORWARD -p udp -d 10.107.115.155 --dport 53 -j ACCEPT sudo iptables -A FORWARD -p tcp -d 10.107.115.33 --dport 5432 -j ACCEPT sudo iptables -A FORWARD -p tcp -s 10.107.115.33 --sport 5432 -j ACCEPT sudo iptables -A FORWARD -p tcp -d 10.107.115.189 --dport 80 -j ACCEPT sudo iptables -A FORWARD -p tcp -d 10.107.115.189 --dport 443 -j ACCEPT sudo iptables -A FORWARD -p tcp -s 10.107.115.189 -j ACCEPT sudo iptables -A FORWARD -p tcp -d 10.107.115.200 -j ACCEPT sudo iptables -A FORWARD -p tcp -s 10.107.115.200 -j ACCEPT # ================== # POSTROUTING MASQUERADE (for return traffic) # ================== sudo iptables -t nat -A POSTROUTING -p tcp -d 10.107.115.155 -j MASQUERADE sudo iptables -t nat -A POSTROUTING -p udp -d 10.107.115.155 -j MASQUERADE sudo iptables -t nat -A POSTROUTING -p tcp -d 10.107.115.33 -j MASQUERADE sudo iptables -t nat -A POSTROUTING -p tcp -d 10.107.115.189 -j MASQUERADE sudo iptables -t nat -A POSTROUTING -p tcp -d 10.107.115.200 -j MASQUERADE # ================== # INPUT rules (allow incoming) # ================== sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT sudo iptables -A INPUT -p tcp --dport 4445 -j ACCEPT sudo iptables -A INPUT -p tcp --dport 53 -j ACCEPT sudo iptables -A INPUT -p udp --dport 53 -j ACCEPT # ================== # Save rules persistently # ================== sudo sh -c 'iptables-save > /etc/iptables/rules.v4' ``` ### IMPORTANT RULES 1. **NEVER use socat** - It causes port conflicts and doesn't integrate with iptables NAT 2. **NEVER use Incus proxy devices** - They conflict with iptables NAT rules 3. **ALWAYS add OUTPUT rules** - PREROUTING only handles external traffic; local traffic needs OUTPUT 4. **ALWAYS add FORWARD rules** - Without them, traffic won't reach containers 5. **ALWAYS add POSTROUTING MASQUERADE** - Without it, return traffic won't work 6. **ALWAYS set route_localnet=1** - Required for localhost NAT to work ### Testing NAT ```bash # Test from host nc -zv 127.0.0.1 4445 # Should connect to PostgreSQL at 10.107.115.33:5432 # Test from external nc -zv 63.141.255.9 4445 # Should connect to PostgreSQL at 10.107.115.33:5432 # Test DNS dig @127.0.0.1 webmail.pragmatismo.com.br # Should return 63.141.255.9 ``` --- ## CoreDNS Setup ### Corefile Template ```corefile ddsites.com.br:53 { file /opt/gbo/data/ddsites.com.br.zone bind 0.0.0.0 reload 6h acl { allow type ANY net 10.0.0.0/8 127.0.0.0/8 allow type ANY net /32 allow type A net 0.0.0.0/0 allow type AAAA net 0.0.0.0/0 allow type MX net 0.0.0.0/0 allow type TXT net 0.0.0.0/0 allow type NS net 0.0.0.0/0 allow type SOA net 0.0.0.0/0 allow type SRV net 0.0.0.0/0 allow type CNAME net 0.0.0.0/0 allow type HTTPS net 0.0.0.0/0 allow type CAA net 0.0.0.0/0 block } cache errors } pragmatismo.com.br:53 { file /opt/gbo/data/pragmatismo.com.br.zone bind 0.0.0.0 reload 6h acl { allow type ANY net 10.0.0.0/8 127.0.0.0/8 allow type ANY net /32 allow type A net 0.0.0.0/0 allow type AAAA net 0.0.0.0/0 allow type MX net 0.0.0.0/0 allow type TXT net 0.0.0.0/0 allow type NS net 0.0.0.0/0 allow type SOA net 0.0.0.0/0 allow type SRV net 0.0.0.0/0 allow type CNAME net 0.0.0.0/0 allow type HTTPS net 0.0.0.0/0 allow type CAA net 0.0.0.0/0 block } cache errors } . { forward . 8.8.8.8 1.1.1.1 cache errors log } ``` ### Zone File Template (pragmatismo.com.br) ``` $ORIGIN pragmatismo.com.br. $TTL 3600 @ IN SOA ns1.ddsites.com.br. hostmaster.dmeans.info. ( 2026032301 ; Serial (YYYYMMDDNN) 86400 ; Refresh 900 ; Retry 1209600 ; Expire 3600 ; Minimum TTL ) @ IN CAA 0 issue "letsencrypt.org" @ IN CAA 0 issuewild ";" @ IN CAA 0 iodef "mailto:security@pragmatismo.com.br" @ IN HTTPS 1 . alpn="h2,h3" @ IN NS ns1.ddsites.com.br. @ IN NS ns2.ddsites.com.br. @ IN A ns1 IN A ns2 IN A @ IN MX 10 mail.pragmatismo.com.br. mail IN A www IN A webmail IN A drive IN A drive-api IN A alm IN A tables IN A gb IN A gb6 IN A ``` ### Starting CoreDNS in Container ```bash # CoreDNS won't start via systemd in Incus containers by default # Use nohup to start it incus exec dns -- bash -c 'mkdir -p /opt/gbo/logs && nohup /opt/gbo/bin/coredns -conf /opt/gbo/conf/Corefile > /opt/gbo/logs/coredns.log 2>&1 &' ``` ### DNS Zone Records (CRITICAL - Use A records, NOT CNAMEs for internal services) ``` # WRONG - CNAME causes resolution issues webmail IN CNAME mail # CORRECT - Direct A record webmail IN A mail IN A ``` --- **ALWAYS remove socat and Incus proxy devices before configuring iptables NAT:** ```bash # Remove socat pkill -9 -f socat 2>/dev/null rm -f /usr/bin/socat /usr/sbin/socat 2>/dev/null # Remove all proxy devices from all containers for c in $(incus list --format csv -c n); do for d in $(incus config device list $c 2>/dev/null | grep -E 'proxy|port'); do echo "Removing $d from $c" incus config device remove $c $d 2>/dev/null done done ``` --- ## installer.rs Improvements Required ### 1. New Module Structure ``` botserver/src/core/package_manager/ ├── mod.rs ├── component.rs # ComponentConfig (existing) ├── installer.rs # PackageManager (existing) ├── container.rs # NEW: Container deployment logic └── templates/ # NEW: Service file templates ├── dns.service ├── email.service ├── alm.service ├── minio.service └── webmail.service ``` ### 2. Container Settings in ComponentConfig ```rust // Add to component.rs #[derive(Debug, Clone)] pub struct NatRule { pub port: u16, pub protocol: String, // "tcp" or "udp" } #[derive(Debug, Clone)] pub struct ContainerSettings { pub container_name: String, pub ip: String, pub user: String, pub group: Option, pub working_dir: Option, pub service_template: String, pub nat_rules: Vec, pub binary_path: String, // "/opt/gbo/bin/coredns" pub config_path: String, // "/opt/gbo/conf/Corefile" pub data_path: Option, // "/opt/gbo/data" pub exec_cmd_args: Vec, // ["--config", "/opt/gbo/conf/Corefile"] pub internal_ports: Vec, // Ports container listens on internally pub external_port: Option, // External port (if different from internal) } ``` ### 3. Component Registration with Container Settings ```rust fn register_dns(&mut self) { self.components.insert( "dns".to_string(), ComponentConfig { name: "dns".to_string(), // ... existing fields ... // NEW: Container settings container: Some(ContainerSettings { container_name: "dns".to_string(), ip: "10.107.115.155".to_string(), user: "root".to_string(), group: None, working_dir: None, service_template: include_str!("templates/dns.service").to_string(), nat_rules: vec![ NatRule { port: 53, protocol: "tcp".to_string() }, NatRule { port: 53, protocol: "udp".to_string() }, ], binary_path: "/opt/gbo/bin/coredns".to_string(), config_path: "/opt/gbo/conf/Corefile".to_string(), data_path: Some("/opt/gbo/data".to_string()), exec_cmd_args: vec!["-conf".to_string(), "/opt/gbo/conf/Corefile".to_string()], internal_ports: vec![53], external_port: Some(53), }), }, ); } fn register_tables(&mut self) { // PostgreSQL with external port 4445 self.components.insert( "tables".to_string(), ComponentConfig { name: "tables".to_string(), container: Some(ContainerSettings { container_name: "tables".to_string(), ip: "10.107.115.33".to_string(), user: "root".to_string(), nat_rules: vec![ NatRule { port: 4445, protocol: "tcp".to_string() }, ], internal_ports: vec![5432], external_port: Some(4445), // ... }), }, ); } ``` ### 4. Container Deployment Methods ```rust // Add to installer.rs impl PackageManager { /// Bootstrap a container with all its services and NAT rules pub async fn bootstrap_container( &self, container_name: &str, source_lxd: Option<&str>, ) -> Result<()> { info!("Bootstrapping container: {}", container_name); // 0. CLEANUP - Remove any existing socat or proxy devices self.cleanup_existing(container_name).await?; // 1. Copy from source LXD if migrating if let Some(source_remote) = source_lxd { self.copy_container(source_remote, container_name).await?; } // 2. Ensure network is configured self.ensure_network(container_name).await?; // 3. Sync data from host to container self.sync_data_to_container(container_name).await?; // 4. Fix permissions self.fix_permissions(container_name).await?; // 5. Install and start service self.install_systemd_service(container_name).await?; // 6. Configure NAT rules on host (ONLY iptables, never socat) self.configure_iptables_nat(container_name).await?; // 7. Reload DNS if dns container if container_name == "dns" { self.reload_dns_zones().await?; } info!("Container {} bootstrapped successfully", container_name); Ok(()) } /// Cleanup existing socat and proxy devices async fn cleanup_existing(&self, container: &str) -> Result<()> { // Remove socat processes SafeCommand::new("pkill") .and_then(|c| c.arg("-9")) .and_then(|c| c.arg("-f")) .and_then(|c| c.arg("socat")) .execute()?; // Remove proxy devices from container let output = SafeCommand::new("incus") .and_then(|c| c.arg("config")) .and_then(|c| c.arg("device")) .and_then(|c| c.arg("list")) .and_then(|c| c.arg(container)) .and_then(|cmd| cmd.execute_with_output())?; let output_str = String::from_utf8_lossy(&output.stdout); for line in output_str.lines() { if line.contains("proxy") || line.contains("port") { let parts: Vec<&str> = line.split_whitespace().collect(); if let Some(name) = parts.first() { SafeCommand::new("incus") .and_then(|c| c.arg("config")) .and_then(|c| c.arg("device")) .and_then(|c| c.arg("remove")) .and_then(|c| c.arg(container)) .and_then(|c| c.arg(name)) .execute()?; } } } Ok(()) } /// Copy container from LXD source async fn copy_container(&self, source_remote: &str, name: &str) -> Result<()> { info!("Copying container {} from {}", name, source_remote); SafeCommand::new("incus") .and_then(|c| c.arg("copy")) .and_then(|c| c.arg("--instance-only")) .and_then(|c| c.arg(format!("{}:{}", source_remote, name))) .and_then(|c| c.arg(name)) .and_then(|cmd| cmd.execute()) .context("Failed to copy container")?; SafeCommand::new("incus") .and_then(|c| c.arg("start")) .and_then(|c| c.arg(name)) .and_then(|cmd| cmd.execute()) .context("Failed to start container")?; Ok(()) } /// Add eth0 network to container async fn ensure_network(&self, container: &str) -> Result<()> { let output = SafeCommand::new("incus") .and_then(|c| c.arg("config")) .and_then(|c| c.arg("device")) .and_then(|c| c.arg("list")) .and_then(|c| c.arg(container)) .and_then(|cmd| cmd.execute_with_output())?; let output_str = String::from_utf8_lossy(&output.stdout); if !output_str.contains("eth0") { SafeCommand::new("incus") .and_then(|c| c.arg("config")) .and_then(|c| c.arg("device")) .and_then(|c| c.arg("add")) .and_then(|c| c.arg(container)) .and_then(|c| c.arg("eth0")) .and_then(|c| c.arg("nic")) .and_then(|c| c.arg("name=eth0")) .and_then(|c| c.arg("network=PROD-GBO")) .and_then(|cmd| cmd.execute())?; } Ok(()) } /// Sync data from host to container async fn sync_data_to_container(&self, container: &str) -> Result<()> { let source_path = format!( "/opt/gbo/tenants/{}/{}/", self.tenant, container ); if Path::new(&source_path).exists() { info!("Syncing data for {}", container); SafeCommand::new("incus") .and_then(|c| c.arg("exec")) .and_then(|c| c.arg(container)) .and_then(|c| c.arg("--")) .and_then(|c| c.arg("mkdir")) .and_then(|c| c.arg("-p")) .and_then(|c| c.arg("/opt/gbo")) .and_then(|cmd| cmd.execute())?; SafeCommand::new("incus") .and_then(|c| c.arg("file")) .and_then(|c| c.arg("push")) .and_then(|c| c.arg("--recursive")) .and_then(|c| c.arg(format!("{}.", source_path))) .and_then(|c| c.arg(format!("{}:/opt/gbo/", container))) .and_then(|cmd| cmd.execute())?; } Ok(()) } /// Fix file permissions based on container user async fn fix_permissions(&self, container: &str) -> Result<()> { let settings = self.get_container_settings(container)?; if let Some(user) = &settings.user { let chown_cmd = if let Some(group) = &settings.group { format!("chown -R {}:{} /opt/gbo/", user, group) } else { format!("chown -R {}:{} /opt/gbo/", user, user) }; SafeCommand::new("incus") .and_then(|c| c.arg("exec")) .and_then(|c| c.arg(container)) .and_then(|c| c.arg("--")) .and_then(|c| c.arg("sh")) .and_then(|c| c.arg("-c")) .and_then(|c| c.arg(&chown_cmd)) .and_then(|cmd| cmd.execute())?; } // Make binaries executable SafeCommand::new("incus") .and_then(|c| c.arg("exec")) .and_then(|c| c.arg(container)) .and_then(|c| c.arg("--")) .and_then(|c| c.arg("chmod")) .and_then(|c| c.arg("+x")) .and_then(|c| c.arg(format!("{}/bin/*", self.base_path.display()))) .and_then(|cmd| cmd.execute())?; Ok(()) } /// Install systemd service file and start async fn install_systemd_service(&self, container: &str) -> Result<()> { let settings = self.get_container_settings(container)?; let service_name = format!("{}.service", container); let temp_path = format!("/tmp/{}", service_name); std::fs::write(&temp_path, &settings.service_template) .context("Failed to write service template")?; SafeCommand::new("incus") .and_then(|c| c.arg("file")) .and_then(|c| c.arg("push")) .and_then(|c| c.arg(&temp_path)) .and_then(|c| c.arg(format!("{}:/etc/systemd/system/{}", container, service_name))) .and_then(|cmd| cmd.execute())?; for cmd_args in [ ["daemon-reload"], &["enable", &service_name], &["start", &service_name], ] { let mut cmd = SafeCommand::new("incus") .and_then(|c| c.arg("exec")) .and_then(|c| c.arg(container)) .and_then(|c| c.arg("--")) .and_then(|c| c.arg("systemctl")); for arg in cmd_args { cmd = cmd.and_then(|c| c.arg(arg)); } cmd.execute()?; } std::fs::remove_file(&temp_path).ok(); Ok(()) } /// Configure iptables NAT rules on host - ONLY method allowed, NEVER socat async fn configure_iptables_nat(&self, container: &str) -> Result<()> { let settings = self.get_container_settings(container)?; // Set route_localnet if not already set SafeCommand::new("sudo") .and_then(|c| c.arg("sysctl")) .and_then(|c| c.arg("-w")) .and_then(|c| c.arg("net.ipv4.conf.all.route_localnet=1")) .execute()?; for rule in &settings.nat_rules { // PREROUTING rule - for external traffic SafeCommand::new("sudo") .and_then(|c| c.arg("iptables")) .and_then(|c| c.arg("-t")) .and_then(|c| c.arg("nat")) .and_then(|c| c.arg("-A")) .and_then(|c| c.arg("PREROUTING")) .and_then(|c| c.arg("-p")) .and_then(|c| c.arg(&rule.protocol)) .and_then(|c| c.arg("--dport")) .and_then(|c| c.arg(rule.port.to_string())) .and_then(|c| c.arg("-j")) .and_then(|c| c.arg("DNAT")) .and_then(|c| c.arg("--to-destination")) .and_then(|c| c.arg(format!("{}:{}", settings.ip, rule.port))) .and_then(|cmd| cmd.execute())?; // OUTPUT rule - for local traffic (CRITICAL for NAT to work) SafeCommand::new("sudo") .and_then(|c| c.arg("iptables")) .and_then(|c| c.arg("-t")) .and_then(|c| c.arg("nat")) .and_then(|c| c.arg("-A")) .and_then(|c| c.arg("OUTPUT")) .and_then(|c| c.arg("-p")) .and_then(|c| c.arg(&rule.protocol)) .and_then(|c| c.arg("--dport")) .and_then(|c| c.arg(rule.port.to_string())) .and_then(|c| c.arg("-j")) .and_then(|c| c.arg("DNAT")) .and_then(|c| c.arg("--to-destination")) .and_then(|c| c.arg(format!("{}:{}", settings.ip, rule.port))) .and_then(|cmd| cmd.execute())?; // FORWARD rules SafeCommand::new("sudo") .and_then(|c| c.arg("iptables")) .and_then(|c| c.arg("-A")) .and_then(|c| c.arg("FORWARD")) .and_then(|c| c.arg("-p")) .and_then(|c| c.arg(&rule.protocol)) .and_then(|c| c.arg("-d")) .and_then(|c| c.arg(&settings.ip)) .and_then(|c| c.arg("--dport")) .and_then(|c| c.arg(rule.port.to_string())) .and_then(|c| c.arg("-j")) .and_then(|c| c.arg("ACCEPT")) .and_then(|cmd| cmd.execute())?; } // POSTROUTING MASQUERADE for return traffic SafeCommand::new("sudo") .and_then(|c| c.arg("iptables")) .and_then(|c| c.arg("-t")) .and_then(|c| c.arg("nat")) .and_then(|c| c.arg("-A")) .and_then(|c| c.arg("POSTROUTING")) .and_then(|c| c.arg("-p")) .and_then(|c| c.arg("tcp")) .and_then(|c| c.arg("-d")) .and_then(|c| c.arg(&settings.ip)) .and_then(|c| c.arg("-j")) .and_then(|c| c.arg("MASQUERADE")) .and_then(|cmd| cmd.execute())?; // Save rules SafeCommand::new("sudo") .and_then(|c| c.arg("sh")) .and_then(|c| c.arg("-c")) .and_then(|c| c.arg("iptables-save > /etc/iptables/rules.v4")) .and_then(|cmd| cmd.execute())?; Ok(()) } /// Start CoreDNS (special case - doesn't work well with systemd in Incus) async fn start_coredns(&self, container: &str) -> Result<()> { SafeCommand::new("incus") .and_then(|c| c.arg("exec")) .and_then(|c| c.arg(container)) .and_then(|c| c.arg("--")) .and_then(|c| c.arg("bash")) .and_then(|c| c.arg("-c")) .and_then(|c| c.arg("mkdir -p /opt/gbo/logs && nohup /opt/gbo/bin/coredns -conf /opt/gbo/conf/Corefile > /opt/gbo/logs/coredns.log 2>&1 &")) .and_then(|cmd| cmd.execute())?; Ok(()) } /// Reload DNS zones with new IPs async fn reload_dns_zones(&self) -> Result<()> { // Update zone files to point to new IP SafeCommand::new("incus") .and_then(|c| c.arg("exec")) .and_then(|c| c.arg("dns")) .and_then(|c| c.arg("--")) .and_then(|c| c.arg("sh")) .and_then(|c| c.arg("-c")) .and_then(|c| c.arg("sed -i 's/OLD_IP/NEW_IP/g' /opt/gbo/data/*.zone")) .and_then(|cmd| cmd.execute())?; // Restart coredns self.start_coredns("dns").await?; Ok(()) } /// Get container settings for a component fn get_container_settings(&self, container: &str) -> Result<&ContainerSettings> { self.components .get(container) .and_then(|c| c.container.as_ref()) .context("Container settings not found") } } ``` ### 5. Binary Installation (For Fresh Containers) ```rust /// Install binary to container from URL or fallback async fn install_binary_to_container( &self, container: &str, component: &str, ) -> Result<()> { let config = self.components.get(component) .context("Component not found")?; let binary_name = config.binary_name.as_ref() .context("No binary name")?; let settings = config.container.as_ref() .context("No container settings")?; // Check if already exists let check = SafeCommand::new("incus") .and_then(|c| c.arg("exec")) .and_then(|c| c.arg(container)) .and_then(|c| c.arg("--")) .and_then(|c| c.arg("test")) .and_then(|c| c.arg("-f")) .and_then(|c| c.arg(&settings.binary_path)) .and_then(|cmd| cmd.execute()); if check.is_ok() { info!("Binary {} already exists in {}", binary_name, container); return Ok(()); } // Download if URL available if let Some(url) = &config.download_url { self.download_and_push_binary(container, url, binary_name).await?; } // Make executable SafeCommand::new("incus") .and_then(|c| c.arg("exec")) .and_then(|c| c.arg(container)) .and_then(|c| c.arg("--")) .and_then(|c| c.arg("chmod")) .and_then(|c| c.arg("+x")) .and_then(|c| c.arg(&settings.binary_path)) .and_then(|cmd| cmd.execute())?; Ok(()) } ``` --- ## Full Bootstrap API ```rust /// Bootstrap an entire tenant pub async fn bootstrap_tenant( state: &AppState, tenant: &str, containers: &[&str], source_remote: Option<&str>, ) -> Result<()> { let pm = PackageManager::new(InstallMode::Container, Some(tenant.to_string()))?; for container in containers { pm.bootstrap_container(container, source_remote).await?; } info!("Tenant {} bootstrapped successfully", tenant); Ok(()) } /// Bootstrap all pragmatismo containers pub async fn bootstrap_pragmatismo(state: &AppState) -> Result<()> { let containers = [ "dns", "email", "webmail", "alm", "drive", "tables", "system", "proxy", "alm-ci", "table-editor" ]; bootstrap_tenant(state, "pragmatismo", &containers, Some("lxd-source")).await } ``` --- ## Command Line Usage ```bash # Bootstrap single container cargo run --bin bootstrap -- container dns --tenant pragmatismo # Bootstrap all containers for a tenant cargo run --bin bootstrap -- tenant pragmatismo --source lxd-source # Only sync data (no copy from LXD) cargo run --bin bootstrap -- sync-data dns --tenant pragmatismo # Only configure NAT cargo run --bin bootstrap -- configure-nat --container dns # Only install service cargo run --bin bootstrap -- install-service dns # Clean up socat and proxy devices cargo run --bin bootstrap -- cleanup --container dns ``` --- ## Files to Create/Modify ### New Files 1. `botserver/src/core/package_manager/container.rs` - Container deployment logic 2. `botserver/src/core/package_manager/templates/dns.service` 3. `botserver/src/core/package_manager/templates/email.service` 4. `botserver/src/core/package_manager/templates/alm.service` 5. `botserver/src/core/package_manager/templates/minio.service` 6. `botserver/src/core/package_manager/templates/webmail.service` 7. `botserver/src/core/package_manager/templates/tables-postgresql.service` ### Modified Files 1. `botserver/src/core/package_manager/component.rs` - Add ContainerSettings 2. `botserver/src/core/package_manager/installer.rs` - Add container methods, update registrations --- ## Testing Checklist After implementation, verify: - [ ] `incus list` shows all containers running - [ ] `nc -zv 127.0.0.1 4445` - PostgreSQL accessible - [ ] `dig @127.0.0.1 webmail.pragmatismo.com.br` - Returns correct IP - [ ] `curl https://webmail.pragmatismo.com.br` - Webmail accessible - [ ] NAT rules work from external IP - [ ] Zone files have correct A records (not CNAMEs) - [ ] Services survive container restart - [ ] `which socat` returns nothing on host - [ ] No proxy devices in any container --- ## Known Issues Fixed 1. **socat conflicts with iptables** - NEVER use socat, use ONLY iptables NAT 2. **Incus proxy devices conflict with NAT** - Remove all proxy devices before setting up NAT 3. **PREROUTING doesn't handle local traffic** - Must add OUTPUT rules 4. **CoreDNS won't start via systemd in Incus** - Use nohup instead 5. **DNS CNAME resolution issues** - Use A records for internal services 6. **route_localnet needed for localhost NAT** - Set sysctl before NAT rules 7. **FORWARD chain blocks container traffic** - Must add FORWARD ACCEPT rules 8. **Return traffic fails without MASQUERADE** - Add POSTROUTING MASQUERADE rules 9. **Binary permissions** - chmod +x after push 10. **Apache SSL needs mod_ssl enabled** - Run `a2enmod ssl` before starting Apache