753 lines
14 KiB
Markdown
753 lines
14 KiB
Markdown
# K3s + Caddy + Traefik 部署文档
|
||
|
||
## 架构概述
|
||
|
||
```
|
||
互联网
|
||
↓
|
||
Caddy (宿主机 80/443) - SSL 终止 + 自动证书
|
||
↓
|
||
Traefik (ClusterIP) - Ingress 路由
|
||
↓
|
||
应用 Services
|
||
↓
|
||
应用 Pods
|
||
```
|
||
|
||
**职责分工：**
|
||
- **Caddy**: SSL/TLS 终止，自动获取和续期 Let's Encrypt 证书
|
||
- **Traefik**: Kubernetes Ingress 控制器，根据 Ingress 规则路由流量
|
||
- **应用**: 通过 Service 和 Ingress 暴露服务
|
||
|
||
---
|
||
|
||
## 前置要求
|
||
|
||
### 系统要求
|
||
- Linux 系统（Ubuntu/Debian/CentOS 等）
|
||
- 至少 2GB RAM
|
||
- sudo 权限
|
||
- 域名已解析到服务器 IP
|
||
|
||
### 端口要求
|
||
- 80 (HTTP)
|
||
- 443 (HTTPS)
|
||
- 6443 (K3s API，可选)
|
||
|
||
---
|
||
|
||
## 安装步骤
|
||
|
||
### 1. 安装 K3s
|
||
|
||
```bash
|
||
# 下载并安装 k3s
|
||
curl -sfL https://get.k3s.io -o /tmp/k3s-install.sh
|
||
sudo sh /tmp/k3s-install.sh
|
||
|
||
# 验证安装
|
||
sudo kubectl get nodes
|
||
```
|
||
|
||
**预期输出：**
|
||
```
|
||
NAME STATUS ROLES AGE VERSION
|
||
host Ready control-plane 10s v1.34.3+k3s1
|
||
```
|
||
|
||
### 2. 配置 Traefik 为 ClusterIP
|
||
|
||
K3s 默认安装 Traefik 为 LoadBalancer 类型，需要改为 ClusterIP 以便 Caddy 使用宿主机端口。
|
||
|
||
```bash
|
||
# 将 Traefik Service 改为 ClusterIP
|
||
sudo kubectl patch svc traefik -n kube-system -p '{"spec":{"type":"ClusterIP"}}'
|
||
|
||
# 验证
|
||
sudo kubectl get svc -n kube-system traefik
|
||
```
|
||
|
||
**预期输出：**
|
||
```
|
||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
|
||
traefik ClusterIP 10.43.x.x <none> 80/TCP,443/TCP
|
||
```
|
||
|
||
### 3. 部署 Caddy
|
||
|
||
创建 Caddy 配置文件 `caddy-deployment.yaml`：
|
||
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: ConfigMap
|
||
metadata:
|
||
name: caddy-config
|
||
namespace: default
|
||
data:
|
||
Caddyfile: |
|
||
{
|
||
email your-email@example.com
|
||
}
|
||
|
||
# 添加你的域名配置
|
||
app1.example.com {
|
||
reverse_proxy traefik.kube-system.svc.cluster.local:80
|
||
}
|
||
|
||
app2.example.com {
|
||
reverse_proxy traefik.kube-system.svc.cluster.local:80
|
||
}
|
||
---
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
metadata:
|
||
name: caddy
|
||
namespace: default
|
||
spec:
|
||
replicas: 1
|
||
selector:
|
||
matchLabels:
|
||
app: caddy
|
||
template:
|
||
metadata:
|
||
labels:
|
||
app: caddy
|
||
spec:
|
||
containers:
|
||
- name: caddy
|
||
image: caddy:latest
|
||
ports:
|
||
- containerPort: 80
|
||
hostPort: 80
|
||
- containerPort: 443
|
||
hostPort: 443
|
||
volumeMounts:
|
||
- name: config
|
||
mountPath: /etc/caddy
|
||
- name: data
|
||
mountPath: /data
|
||
volumes:
|
||
- name: config
|
||
configMap:
|
||
name: caddy-config
|
||
- name: data
|
||
emptyDir: {}
|
||
---
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
name: caddy
|
||
namespace: default
|
||
spec:
|
||
selector:
|
||
app: caddy
|
||
ports:
|
||
- name: http
|
||
port: 80
|
||
targetPort: 80
|
||
- name: https
|
||
port: 443
|
||
targetPort: 443
|
||
type: ClusterIP
|
||
```
|
||
|
||
**部署 Caddy：**
|
||
|
||
```bash
|
||
# 应用配置
|
||
sudo kubectl apply -f caddy-deployment.yaml
|
||
|
||
# 验证 Caddy 运行状态
|
||
sudo kubectl get pods -n default -l app=caddy
|
||
```
|
||
|
||
### 4. 部署测试应用
|
||
|
||
创建测试应用 `test-app.yaml`：
|
||
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: ConfigMap
|
||
metadata:
|
||
name: test-html
|
||
namespace: default
|
||
data:
|
||
index.html: |
|
||
<!DOCTYPE html>
|
||
<html>
|
||
<head>
|
||
<title>K3s Test</title>
|
||
<style>
|
||
body {
|
||
font-family: Arial, sans-serif;
|
||
display: flex;
|
||
justify-content: center;
|
||
align-items: center;
|
||
height: 100vh;
|
||
margin: 0;
|
||
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
|
||
color: white;
|
||
}
|
||
.container {
|
||
text-align: center;
|
||
padding: 40px;
|
||
background: rgba(255, 255, 255, 0.1);
|
||
border-radius: 10px;
|
||
}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<div class="container">
|
||
<h1>K3s + Caddy + Traefik</h1>
|
||
<p>SSL Termination Working!</p>
|
||
</div>
|
||
</body>
|
||
</html>
|
||
---
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
metadata:
|
||
name: test-app
|
||
namespace: default
|
||
spec:
|
||
replicas: 1
|
||
selector:
|
||
matchLabels:
|
||
app: test-app
|
||
template:
|
||
metadata:
|
||
labels:
|
||
app: test-app
|
||
spec:
|
||
containers:
|
||
- name: nginx
|
||
image: nginx:alpine
|
||
ports:
|
||
- containerPort: 80
|
||
volumeMounts:
|
||
- name: html
|
||
mountPath: /usr/share/nginx/html
|
||
volumes:
|
||
- name: html
|
||
configMap:
|
||
name: test-html
|
||
---
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
name: test-app
|
||
namespace: default
|
||
spec:
|
||
selector:
|
||
app: test-app
|
||
ports:
|
||
- port: 80
|
||
targetPort: 80
|
||
type: ClusterIP
|
||
```
|
||
|
||
**部署应用：**
|
||
|
||
```bash
|
||
sudo kubectl apply -f test-app.yaml
|
||
```
|
||
|
||
### 5. 创建 Ingress 规则
|
||
|
||
创建 Ingress 配置 `test-ingress.yaml`：
|
||
|
||
```yaml
|
||
apiVersion: networking.k8s.io/v1
|
||
kind: Ingress
|
||
metadata:
|
||
name: test-app-ingress
|
||
namespace: default
|
||
annotations:
|
||
traefik.ingress.kubernetes.io/router.entrypoints: web
|
||
spec:
|
||
rules:
|
||
- host: app1.example.com
|
||
http:
|
||
paths:
|
||
- path: /
|
||
pathType: Prefix
|
||
backend:
|
||
service:
|
||
name: test-app
|
||
port:
|
||
number: 80
|
||
```
|
||
|
||
**应用 Ingress：**
|
||
|
||
```bash
|
||
sudo kubectl apply -f test-ingress.yaml
|
||
|
||
# 查看 Ingress 状态
|
||
sudo kubectl get ingress -n default
|
||
```
|
||
|
||
---
|
||
|
||
## 验证部署
|
||
|
||
### 1. 检查所有组件状态
|
||
|
||
```bash
|
||
# 查看所有 Pods
|
||
sudo kubectl get pods -A
|
||
|
||
# 查看 Services
|
||
sudo kubectl get svc -A
|
||
|
||
# 查看 Ingress
|
||
sudo kubectl get ingress -A
|
||
```
|
||
|
||
### 2. 测试 HTTP 访问
|
||
|
||
```bash
|
||
# 测试 HTTP（会自动重定向到 HTTPS）
|
||
curl -I http://app1.example.com
|
||
```
|
||
|
||
**预期输出：**
|
||
```
|
||
HTTP/1.1 308 Permanent Redirect
|
||
Location: https://app1.example.com/
|
||
```
|
||
|
||
### 3. 测试 HTTPS 访问
|
||
|
||
```bash
|
||
# 测试 HTTPS
|
||
curl https://app1.example.com
|
||
```
|
||
|
||
**预期输出：** HTML 页面内容
|
||
|
||
### 4. 检查 SSL 证书
|
||
|
||
```bash
|
||
# 查看证书信息
|
||
echo | openssl s_client -connect app1.example.com:443 -servername app1.example.com 2>/dev/null | openssl x509 -noout -dates
|
||
```
|
||
|
||
---
|
||
|
||
## 添加新应用
|
||
|
||
### 步骤 1: 更新 Caddy 配置
|
||
|
||
```bash
|
||
# 编辑 Caddy ConfigMap
|
||
sudo kubectl edit configmap caddy-config -n default
|
||
```
|
||
|
||
在 Caddyfile 中添加新域名：
|
||
|
||
```
|
||
newapp.example.com {
|
||
reverse_proxy traefik.kube-system.svc.cluster.local:80
|
||
}
|
||
```
|
||
|
||
### 步骤 2: 重启 Caddy
|
||
|
||
```bash
|
||
sudo kubectl rollout restart deployment caddy -n default
|
||
```
|
||
|
||
### 步骤 3: 部署应用
|
||
|
||
```bash
|
||
# 创建应用的 Deployment 和 Service
|
||
sudo kubectl apply -f your-app.yaml
|
||
```
|
||
|
||
### 步骤 4: 创建 Ingress
|
||
|
||
```yaml
|
||
apiVersion: networking.k8s.io/v1
|
||
kind: Ingress
|
||
metadata:
|
||
name: newapp-ingress
|
||
namespace: default
|
||
spec:
|
||
rules:
|
||
- host: newapp.example.com
|
||
http:
|
||
paths:
|
||
- path: /
|
||
pathType: Prefix
|
||
backend:
|
||
service:
|
||
name: your-service-name
|
||
port:
|
||
number: 80
|
||
```
|
||
|
||
```bash
|
||
sudo kubectl apply -f newapp-ingress.yaml
|
||
```
|
||
|
||
### 步骤 5: 等待证书获取
|
||
|
||
```bash
|
||
# 查看 Caddy 日志，确认证书获取成功
|
||
sudo kubectl logs -n default -l app=caddy --tail=50
|
||
```
|
||
|
||
---
|
||
|
||
## 常用管理命令
|
||
|
||
### 查看日志
|
||
|
||
```bash
|
||
# Caddy 日志
|
||
sudo kubectl logs -n default -l app=caddy -f
|
||
|
||
# Traefik 日志
|
||
sudo kubectl logs -n kube-system -l app.kubernetes.io/name=traefik -f
|
||
|
||
# 应用日志
|
||
sudo kubectl logs -n default -l app=your-app -f
|
||
```
|
||
|
||
### 重启服务
|
||
|
||
```bash
|
||
# 重启 Caddy
|
||
sudo kubectl rollout restart deployment caddy -n default
|
||
|
||
# 重启应用
|
||
sudo kubectl rollout restart deployment your-app -n default
|
||
```
|
||
|
||
### 查看资源状态
|
||
|
||
```bash
|
||
# 查看所有资源
|
||
sudo kubectl get all,ingress -n default
|
||
|
||
# 查看 ConfigMap
|
||
sudo kubectl get configmap -n default
|
||
|
||
# 查看详细信息
|
||
sudo kubectl describe pod <pod-name> -n default
|
||
```
|
||
|
||
### 更新配置
|
||
|
||
```bash
|
||
# 编辑 Caddy 配置
|
||
sudo kubectl edit configmap caddy-config -n default
|
||
|
||
# 编辑 Ingress
|
||
sudo kubectl edit ingress <ingress-name> -n default
|
||
```
|
||
|
||
---
|
||
|
||
## 故障排查
|
||
|
||
### 问题 1: Caddy Pod 无法启动（Pending 状态）
|
||
|
||
**症状：**
|
||
```bash
|
||
sudo kubectl get pods -n default
|
||
# caddy pod 显示 Pending
|
||
```
|
||
|
||
**原因：** 端口 80/443 被占用
|
||
|
||
**解决方案：**
|
||
```bash
|
||
# 检查端口占用
|
||
sudo ss -tlnp | grep -E ':(80|443)\s'
|
||
|
||
# 如果 Traefik 占用，确保已改为 ClusterIP
|
||
sudo kubectl get svc -n kube-system traefik
|
||
|
||
# 如果是 LoadBalancer，执行：
|
||
sudo kubectl patch svc traefik -n kube-system -p '{"spec":{"type":"ClusterIP"}}'
|
||
```
|
||
|
||
### 问题 2: SSL 证书获取失败
|
||
|
||
**症状：** 访问 HTTPS 时证书错误
|
||
|
||
**检查步骤：**
|
||
|
||
```bash
|
||
# 1. 查看 Caddy 日志
|
||
sudo kubectl logs -n default -l app=caddy --tail=100
|
||
|
||
# 2. 确认域名解析正确
|
||
nslookup your-domain.com
|
||
|
||
# 3. 确认端口 80 可访问（Let's Encrypt 需要）
|
||
curl -I http://your-domain.com
|
||
```
|
||
|
||
**常见原因：**
|
||
- 域名未正确解析到服务器 IP
|
||
- 防火墙阻止 80/443 端口
|
||
- Let's Encrypt 速率限制（每周最多 5 次失败）
|
||
|
||
### 问题 3: 访问域名返回 404
|
||
|
||
**症状：** HTTPS 可访问，但返回 404
|
||
|
||
**检查步骤：**
|
||
|
||
```bash
|
||
# 1. 检查 Ingress 配置
|
||
sudo kubectl get ingress -n default
|
||
sudo kubectl describe ingress <ingress-name> -n default
|
||
|
||
# 2. 检查 Service
|
||
sudo kubectl get svc -n default
|
||
|
||
# 3. 检查 Pod 状态
|
||
sudo kubectl get pods -n default
|
||
|
||
# 4. 测试 Traefik 路由
|
||
sudo kubectl exec -n default deployment/caddy -- \
|
||
curl -H "Host: your-domain.com" http://traefik.kube-system.svc.cluster.local:80
|
||
```
|
||
|
||
**解决方案：**
|
||
- 确认 Ingress 的 host 和 service name 正确
|
||
- 确认 Service selector 匹配 Pod labels
|
||
- 确认 Pod 正在运行
|
||
|
||
### 问题 4: 证书自动续期失败
|
||
|
||
**检查：**
|
||
|
||
```bash
|
||
# 查看证书过期时间
|
||
echo | openssl s_client -connect your-domain.com:443 -servername your-domain.com 2>/dev/null | openssl x509 -noout -dates
|
||
|
||
# 查看 Caddy 日志中的续期信息
|
||
sudo kubectl logs -n default -l app=caddy | grep -i renew
|
||
```
|
||
|
||
**解决方案：**
|
||
- Caddy 会自动续期，通常无需手动干预
|
||
- 如果失败，检查域名解析和端口访问
|
||
- 必要时重启 Caddy: `sudo kubectl rollout restart deployment caddy -n default`
|
||
|
||
---
|
||
|
||
## 高级配置
|
||
|
||
### 使用持久化存储保存证书
|
||
|
||
默认配置使用 `emptyDir`，Pod 重启后证书会丢失。生产环境建议使用持久化存储：
|
||
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: PersistentVolumeClaim
|
||
metadata:
|
||
name: caddy-data
|
||
namespace: default
|
||
spec:
|
||
accessModes:
|
||
- ReadWriteOnce
|
||
resources:
|
||
requests:
|
||
storage: 1Gi
|
||
---
|
||
# 在 Deployment 中使用 PVC
|
||
spec:
|
||
template:
|
||
spec:
|
||
volumes:
|
||
- name: data
|
||
persistentVolumeClaim:
|
||
claimName: caddy-data
|
||
```
|
||
|
||
### 配置 Caddy 日志级别
|
||
|
||
在 Caddyfile 中添加：
|
||
|
||
```
|
||
{
|
||
email your-email@example.com
|
||
log {
|
||
level DEBUG
|
||
}
|
||
}
|
||
```
|
||
|
||
### 配置自定义 TLS 选项
|
||
|
||
```
|
||
your-domain.com {
|
||
tls {
|
||
protocols tls1.2 tls1.3
|
||
ciphers TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
|
||
}
|
||
reverse_proxy traefik.kube-system.svc.cluster.local:80
|
||
}
|
||
```
|
||
|
||
### 配置 Traefik 中间件
|
||
|
||
为 Ingress 添加中间件（如重定向、认证等）：
|
||
|
||
```yaml
|
||
apiVersion: traefik.containo.us/v1alpha1
|
||
kind: Middleware
|
||
metadata:
|
||
name: redirect-https
|
||
namespace: default
|
||
spec:
|
||
redirectScheme:
|
||
scheme: https
|
||
permanent: true
|
||
---
|
||
apiVersion: networking.k8s.io/v1
|
||
kind: Ingress
|
||
metadata:
|
||
name: app-ingress
|
||
annotations:
|
||
traefik.ingress.kubernetes.io/router.middlewares: default-redirect-https@kubernetescrd
|
||
spec:
|
||
rules:
|
||
- host: app.example.com
|
||
http:
|
||
paths:
|
||
- path: /
|
||
pathType: Prefix
|
||
backend:
|
||
service:
|
||
name: app-service
|
||
port:
|
||
number: 80
|
||
```
|
||
|
||
---
|
||
|
||
## 安全建议
|
||
|
||
1. **限制 K3s API 访问**
|
||
```bash
|
||
# 仅允许本地访问
|
||
sudo kubectl config set-cluster default --server=https://127.0.0.1:6443
|
||
```
|
||
|
||
2. **配置防火墙**
|
||
```bash
|
||
# 仅开放必要端口
|
||
sudo ufw allow 80/tcp
|
||
sudo ufw allow 443/tcp
|
||
sudo ufw allow 22/tcp
|
||
sudo ufw enable
|
||
```
|
||
|
||
3. **定期更新**
|
||
```bash
|
||
# 更新 K3s
|
||
curl -sfL https://get.k3s.io | sudo sh -
|
||
|
||
# 更新 Caddy
|
||
sudo kubectl set image deployment/caddy caddy=caddy:latest -n default
|
||
```
|
||
|
||
4. **备份证书数据**
|
||
```bash
|
||
# 如果使用 PVC，定期备份
|
||
sudo kubectl exec -n default deployment/caddy -- tar czf - /data > caddy-backup.tar.gz
|
||
```
|
||
|
||
---
|
||
|
||
## 性能优化
|
||
|
||
### 1. 增加 Caddy 副本数（可选）
|
||
|
||
```bash
|
||
sudo kubectl scale deployment caddy --replicas=2 -n default
|
||
```
|
||
|
||
**注意：** 使用 hostPort 时，每个节点只能运行一个副本。
|
||
|
||
### 2. 配置资源限制
|
||
|
||
```yaml
|
||
spec:
|
||
template:
|
||
spec:
|
||
containers:
|
||
- name: caddy
|
||
resources:
|
||
requests:
|
||
memory: "128Mi"
|
||
cpu: "100m"
|
||
limits:
|
||
memory: "256Mi"
|
||
cpu: "500m"
|
||
```
|
||
|
||
### 3. 启用 HTTP/3
|
||
|
||
Caddy 默认启用 HTTP/3，确保 UDP 443 端口开放：
|
||
|
||
```bash
|
||
sudo ufw allow 443/udp
|
||
```
|
||
|
||
---
|
||
|
||
## 卸载
|
||
|
||
```bash
|
||
# 删除应用
|
||
sudo kubectl delete -f test-app.yaml
|
||
sudo kubectl delete -f test-ingress.yaml
|
||
|
||
# 删除 Caddy
|
||
sudo kubectl delete -f caddy-deployment.yaml
|
||
|
||
# 卸载 K3s
|
||
sudo /usr/local/bin/k3s-uninstall.sh
|
||
```
|
||
|
||
---
|
||
|
||
## 参考资源
|
||
|
||
- [K3s 官方文档](https://docs.k3s.io/)
|
||
- [Caddy 官方文档](https://caddyserver.com/docs/)
|
||
- [Traefik 官方文档](https://doc.traefik.io/traefik/)
|
||
- [Kubernetes Ingress 文档](https://kubernetes.io/docs/concepts/services-networking/ingress/)
|
||
|
||
---
|
||
|
||
## 总结
|
||
|
||
这个架构的优势：
|
||
|
||
✅ **自动化 SSL 管理** - Caddy 自动获取和续期证书
|
||
✅ **灵活的路由** - Traefik 提供强大的 Ingress 功能
|
||
✅ **简单部署** - 一键部署，无需手动配置证书
|
||
✅ **高可用** - 支持多副本和负载均衡
|
||
✅ **易于维护** - 声明式配置，版本控制友好
|
||
|
||
适用场景：
|
||
- 个人项目和小型应用
|
||
- 开发和测试环境
|
||
- 需要快速部署的场景
|
||
- 多域名多应用托管
|
||
|
||
---
|
||
|
||
**文档版本：** 1.0
|
||
**最后更新：** 2026-01-29
|
||
**适用于：** K3s v1.34+, Caddy v2.8+, Traefik v3.5+
|