Traefik 另类的服务暴露方式

最近准备重新折腾一下 Kubernetes 的服务暴露方式，以前的方式是彻底剥离 Kubenretes 本身的服务发现，然后改动应用实现 应用+Consul+Fabio 的服务暴露方式；总感觉这种方式不算优雅，所以折腾了一下 Traefik，试了下效果还不错，以下记录了使用 Traefik 的新的服务暴露方式(本文仅针对 HTTP 协议)；

一、Traefik 服务暴露方案

1.1、以前的 Consul+Fabio 方案

以前的服务暴露方案是修改应用代码，使其能对接 Consul，然后 Consul 负责健康监测，检测通过后 Fabio 负责读取，最终上层 Nginx 将流量打到 Fabio 上，Fabio 再将流量路由到健康的 Pod 上；总体架构如下

consul+fabio

这种架构目前有几点不太好的地方，首先是必须应用能成功集成 Consul，需要动应用代码不通用；其次组件过多增加维护成本，尤其是调用链日志不好追踪；这里面需要吐槽下 Consul 和 Fabio，Consul 的集群设计模式要想做到完全的 HA 那么需要在每个 pod 中启动一个 agent，因为只要这个 agent 挂了那么集群认为其上所有注册服务都挂了，这点很恶心人；而 Fabio 的日志目前好像还是不支持合理的输出，好像只能 stdout；目前来看不论是组件复杂度还是维护成本都不怎么友好

1.2、新的 Traefik 方案

使用 Traefik 首先想到就是直接怼 Ingress，这个确实方便也简单；但是在集群 kube-proxy 不走 ipvs 的情况下 iptables 性能确实堪忧；虽说 Traefik 会直连 Pod，但是你 Ingress 暴露 80、443 端口在本机没有对应 Ingress Controller 的情况下还是会走一下 iptables；不论是换 kube-router、kube-proxy 走 ipvs 都不是我想要的，我们需要一种完全远离 Kubernetes Service 的新路子；在看 Traefik 文档的时候，其中说明了 Traefik 只利用 Kubernetes 的 API 来读取相关后端数据，那么我们就可以以此来使用如下的套路

traefik

这个套路的方案很简单，将 Traefik 部署在物理机上，让其直连 Kubernets api 以读取 Ingress 配置和 Pod IP 等信息，然后在这几台物理机上部署好 Kubernetes 的网络组件使其能直连 Pod IP；这种方案能够让流量经过 Traefik 直接路由到后端 Pod，健康检测还是由集群来做；由于 Traefik 连接 Kubernetes api 需要获取一些数据；所以在集群内还是像往常一样创建 Ingress，只不过此时我们并没有 Ingress Controller；这样避免了经过 iptables 转发，不占用全部集群机器的 80、443 端口，同时还能做到高可控

二、Traefik 部署

部署之前首先需要有一个正常访问的集群，然后在另外几台机器上部署 Kubernetes 的网络组件；最终要保证另外几台机器能够直接连通 Pod 的 IP，我这里偷懒直接在 Kubernetes 的其中几个 Node 上部署 Traefik

2.1、Docker Compose

Traefik 的 Docker Compose 如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
version: '3.5'

services:
  traefik:
    image: traefik:v1.6.1-alpine
    container_name: traefik
    command: --configFile=/etc/traefik.toml
    ports:
      - "2080:2080"
      - "2180:2180"
    volumes:
      - ./traefik.toml:/etc/traefik.toml
      - ./k8s-root-ca.pem:/etc/kubernetes/ssl/k8s-root-ca.pem
      - ./log:/var/log/traefik

由于 Kubernetes 集群开启了 RBAC 认证同时采用 TLS 通讯，所以需要挂载 Kubernetes CA 证书，还需要为 Traefik 创建对应的 RBAC 账户以使其能够访问 Kubernetes API

2.2、创建 RBAC 账户

Traefik 连接 Kubernetes API 时需要使用 Service Account 的 Token，Service Account 以及 ClusterRole 等配置具体见 官方文档，下面是我从当前版本的文档中 Copy 出来的

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

创建好以后需要提取 Service Account 的 Token 方便下面使用，提取命令如下

1
kubectl describe secret -n kube-system $(kubectl get secrets -n kube-system | grep traefik-ingress-controller | cut -f1 -d ' ') | grep -E '^token'

2.3、创建 Traefik 配置

Traefik 的具体配置细节请参考 官方文档，以下仅给出一个样例配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
# DEPRECATED - for general usage instruction see [lifeCycle.graceTimeOut].
#
# If both the deprecated option and the new one are given, the deprecated one
# takes precedence.
# A value of zero is equivalent to omitting the parameter, causing
# [lifeCycle.graceTimeOut] to be effective. Pass zero to the new option in
# order to disable the grace period.
#
# Optional
# Default: "0s"
#
# graceTimeOut = "10s"

# Enable debug mode.
# This will install HTTP handlers to expose Go expvars under /debug/vars and
# pprof profiling data under /debug/pprof.
# The log level will be set to DEBUG unless `logLevel` is specified.
#
# Optional
# Default: false
#
# debug = true

# Periodically check if a new version has been released.
#
# Optional
# Default: true
#
checkNewVersion = false

# Backends throttle duration.
#
# Optional
# Default: "2s"
#
# providersThrottleDuration = "2s"

# Controls the maximum idle (keep-alive) connections to keep per-host.
#
# Optional
# Default: 200
#
# maxIdleConnsPerHost = 200

# If set to true invalid SSL certificates are accepted for backends.
# This disables detection of man-in-the-middle attacks so should only be used on secure backend networks.
#
# Optional
# Default: false
#
# insecureSkipVerify = true

# Register Certificates in the rootCA.
#
# Optional
# Default: []
#
# rootCAs = [ "/mycert.cert" ]

# Entrypoints to be used by frontends that do not specify any entrypoint.
# Each frontend can specify its own entrypoints.
#
# Optional
# Default: ["http"]
#
# defaultEntryPoints = ["http", "https"]

# Allow the use of 0 as server weight.
# - false: a weight 0 means internally a weight of 1.
# - true: a weight 0 means internally a weight of 0 (a server with a weight of 0 is removed from the available servers).
#
# Optional
# Default: false
#
# AllowMinWeightZero = true

logLevel = "INFO"

[traefikLog]
  filePath = "/var/log/traefik/traefik.log"
  format   = "json"

[accessLog]
  filePath = "/var/log/traefik/access.log"
  format = "json"

  [accessLog.filters]
    statusCodes = ["200-511"]
    retryAttempts = true

#  [accessLog.fields]
#    defaultMode = "keep"
#    [accessLog.fields.names]
#      "ClientUsername" = "drop"
#      # ...
#
#    [accessLog.fields.headers]
#      defaultMode = "keep"
#      [accessLog.fields.headers.names]
#        "User-Agent" = "redact"
#        "Authorization" = "drop"
#        "Content-Type" = "keep"
#        # ...


[entryPoints]
  [entryPoints.http]
    address = ":2080"
    compress = true
  [entryPoints.traefik]
    address = ":2180"
    compress = true

#    [entryPoints.http.whitelist]
#      sourceRange = ["192.168.1.0/24"]
#      useXForwardedFor = true

#    [entryPoints.http.tls]
#      minVersion = "VersionTLS12"
#      cipherSuites = [
#        "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
#        "TLS_RSA_WITH_AES_256_GCM_SHA384"
#       ]
#      [[entryPoints.http.tls.certificates]]
#        certFile = "path/to/my.cert"
#        keyFile = "path/to/my.key"
#      [[entryPoints.http.tls.certificates]]
#        certFile = "path/to/other.cert"
#        keyFile = "path/to/other.key"
#      # ...
#      [entryPoints.http.tls.clientCA]
#        files = ["path/to/ca1.crt", "path/to/ca2.crt"]
#        optional = false
#
#    [entryPoints.http.redirect]
#      entryPoint = "https"
#      regex = "^http://localhost/(.*)"
#      replacement = "http://mydomain/$1"
#      permanent = true
#
#    [entryPoints.http.auth]
#      headerField = "X-WebAuth-User"
#      [entryPoints.http.auth.basic]
#        users = [
#          "test:$apr1$H6uskkkW$IgXLP6ewTrSuBkTrqE8wj/",
#          "test2:$apr1$d9hr9HBB$4HxwgUir3HP4EsggP/QNo0",
#        ]
#        usersFile = "/path/to/.htpasswd"
#      [entryPoints.http.auth.digest]
#        users = [
#          "test:traefik:a2688e031edb4be6a3797f3882655c05",
#          "test2:traefik:518845800f9e2bfb1f1f740ec24f074e",
#        ]
#        usersFile = "/path/to/.htdigest"
#      [entryPoints.http.auth.forward]
#        address = "https://authserver.com/auth"
#        trustForwardHeader = true
#        [entryPoints.http.auth.forward.tls]
#          ca =  [ "path/to/local.crt"]
#          caOptional = true
#          cert = "path/to/foo.cert"
#          key = "path/to/foo.key"
#          insecureSkipVerify = true
#
#    [entryPoints.http.proxyProtocol]
#      insecure = true
#      trustedIPs = ["10.10.10.1", "10.10.10.2"]
#
#    [entryPoints.http.forwardedHeaders]
#      trustedIPs = ["10.10.10.1", "10.10.10.2"]
#
#  [entryPoints.https]
#    # ...

################################################################
# Kubernetes Ingress configuration backend
################################################################

# Enable Kubernetes Ingress configuration backend.
[kubernetes]

# Kubernetes server endpoint.
#
# Optional for in-cluster configuration, required otherwise.
# Default: empty
#
endpoint = "https://172.16.0.36:6443"

# Bearer token used for the Kubernetes client configuration.
#
# Optional
# Default: empty
#
token = "eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJ0cmFlZmlrLWluZ3Jlc3MtY29udHJvbGxlci10b2tlbi1zbm5iNSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJ0cmFlZmlrLWlyZ3Jlc3MtY29udHJvbGxlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImE4NmI3YWEzLTVmNjQtMTFlOC1hZjYxLWM4MWY2NmUwMzRhNyIsInN1YiI6InN5c3RlbTpxZXJ2aWNlYWNjb3VudDPrdWJlLXN5c3RlbTp0cmFlZmlrLWluZ3Jlc3MtY29udHJvbGxlciJ9.vOFEITuANWGnkER8gukWkTs54BmHXqNpzM55bOb5qXPmI3pZsbei3gtE6tZoqME9P5Lb85cav-8mGZJcoQqqxNBkZJ1YRqy_1O9Apkxa4jA68ipe_NB3L5-exH5cEIrU8iql_r7ycDaKwzsMnAWGPolp1dRkF31u5u8g68oLwF3GR8Z5g4_tLJlTvA53doX7k6Wd6vUygTS3EaQ_qvfXwbcIeaSdWWo2Mym6O0CvIap4jH2w21MbredGURqkRlXEPezKAgRVkr75CdvuvwORnT8YxFLVwuAJs70V-13Ib9v6HAK64GmzcqkAuJtZT8NZKl8Y4TfRGl2_RMq2tk86gD4ShDMedcrto44ZUYHQccsSlpaW5PsN2KBBNPN0-6ca3jIpOmnJojAFUYGM42Wymnx9_4XwHUeeA18-RrercmOaRMdlNq8BzBomAxQB99TqUzRIqpe6m5OotXvouCUnE7qjMwRWmQ5LHjqUGEw_A1pHcalFXQZK0sOCaJOJZIJbc_8rVX-4uxkCBxoIXmzjq8x5a_xPsN4L0aWifkP6co--agw3kOT0O6my8T_CbcZGO9e3OqYPdT4FSl92XlXW8EXHdDpCUJ10aoqJGG2vZSud7IoDxkcScpkj3n6TvyvSRVtk3CtYiIYBpgi7-X2JKkun1a7yFpLogyazz9VlUE4"

# Path to the certificate authority file.
# Used for the Kubernetes client configuration.
#
# Optional
# Default: empty
#
certAuthFilePath = "/etc/kubernetes/ssl/k8s-root-ca.pem"

# Array of namespaces to watch.
#
# Optional
# Default: all namespaces (empty array).
#
namespaces = ["default"]

# Ingress label selector to filter Ingress objects that should be processed.
#
# Optional
# Default: empty (process all Ingresses)
#
# labelselector = "A and not B"

# Value of `kubernetes.io/ingress.class` annotation that identifies Ingress objects to be processed.
# If the parameter is non-empty, only Ingresses containing an annotation with the same value are processed.
# Otherwise, Ingresses missing the annotation, having an empty value, or the value `traefik` are processed.
#
# Note : `ingressClass` option must begin with the "traefik" prefix.
#
# Optional
# Default: empty
#
# ingressClass = "traefik-internal"

# Disable PassHost Headers.
#
# Optional
# Default: false
#
# disablePassHostHeaders = true

# Enable PassTLSCert Headers.
#
# Optional
# Default: false
#
# enablePassTLSCert = true

# Override default configuration template.
#
# Optional
# Default: <built-in template>
#
# filename = "kubernetes.tmpl"


# API definition
[api]
  # Name of the related entry point
  #
  # Optional
  # Default: "traefik"
  #
  entryPoint = "traefik"

  # Enabled Dashboard
  #
  # Optional
  # Default: true
  #
  dashboard = true

  # Enable debug mode.
  # This will install HTTP handlers to expose Go expvars under /debug/vars and
  # pprof profiling data under /debug/pprof.
  # Additionally, the log level will be set to DEBUG.
  #
  # Optional
  # Default: false
  #
  debug = false

# Ping definition
#[ping]
#  # Name of the related entry point
#  #
#  # Optional
#  # Default: "traefik"
#  #
#  entryPoint = "traefik"

2.4、启动 Traefik

所有文件准备好以后直接执行 docker-compose up -d 启动即可，所有文件目录结构如下

1
2
3
4
5
6
7
8
traefik
├── docker-compose.yaml
├── k8s-root-ca.pem
├── log
│   ├── access.log
│   └── traefik.log
├── rbac.yaml
└── traefik.toml

启动成功后可以访问 http://IP:2180 查看 Traefik 的控制面板

dashboard

三、增加 Ingress 配置并测试

3.1、增加 Ingress 配置

虽然这种部署方式脱离了 Kubernetes 的 Service 与 Ingress 负载，但是 Traefik 还是需要通过 Kubernetes 的 Ingress 配置来确定后端负载规则，所以 Ingress 对象我们仍需照常创建；以下为一个 Demo 项目的 deployment、service、ingress 配置示例

  • demo.deploy.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 5
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
      - name: demo
        image: mritd/demo
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
  • demo.svc.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: v1
kind: Service
metadata:
  name: demo
  labels:
    svc: demo
spec:
  ports:
  - port: 8080
    name: http
    targetPort: 80
  selector:
    app: demo
  • demo.ing.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo
  annotations:
    traefik.ingress.kubernetes.io/preserve-host: "true"
spec:
  rules:
  - host: demo.mritd.me
    http:
      paths:
      - backend:
          serviceName: demo
          servicePort: 8080

3.2、测试访问

部署好后应当能从 Traefik 的 Dashboard 中看到新增的 demo ingress，如下所示

dashboard-demo

最后我们使用 curl 测试即可

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 在不使用 Host 头的情况下 Traefik 会返 404(Traefik 根据 Host 路由后端，具体配置参考官方文档)
test36.node ➜  ~ curl http://172.16.0.36:2080
404 page not found

# 指定 Host 头来路由到 demo 的相关 Pod
test36.node ➜  ~ curl -H "Host: demo.mritd.me" http://172.16.0.36:2080
<!DOCTYPE html>
<html lang="zh">
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <title>Running!</title>
    <style type="text/css">
        body {
            width: 100%;
            min-height: 100%;
            background: linear-gradient(to bottom, #fff 0, #b8edff 50%, #83dfff 100%);
            background-attachment: fixed;
        }
    </style>
</head>
<body class=" hasGoogleVoiceExt">
<div align="center">
    <h1>Your container is running!</h1>
    <img src="./docker.png" alt="docker">
</div>
</body>
</html>#

四、其他说明

写这篇文章的目的是给予一种新的服务暴露思路，这篇文章的某些配置并不适合生产使用；生产环境尽量使用独立的机器部署 Traefik，同时最好宿主机二进制方式部署；应用的 Deployment 也应当加入健康检测以防止错误的流量路由；至于 Traefik 的具体细节配置，比如访问日志、Entrypoints 配置、如何连接 Kubernets HA api 等不在本文范畴内，请自行查阅文档；

最后说一下，关于 Traefik 的 HA 只需要部署多个实例即可，还有 Traefik 本身不做日志滚动等，需要自行处理一下日志。

Kubernetes
Kubernetes

本博客所有文章除特别声明外，均采用 CC BY-NC-SA 4.0 国际许可协议进行许可，转载请注明出处。