Jevic

kubernetes 高可用集群api-server请求异常

2019-03-08T20:10:46+08:00

高可用kubernetes 集群下关于使用nginx代理服务连接超时错误处理

报错信息如下图:
issues

I’ve checked rancher nginx-proxy config and it indeed reports proxy_timeout 5;. When I change the config in-place (setting proxy_timeout to 500) and reload nginx, it works OK.

通过上面的问题描述可以得出:

测试发现直连api-server 接口时服务正常。

由于nginx-proxy proxy_timeout 默认超时时间太短,导致了连接api-server 超时请求失败;

所以需要将此 proxy_timeout 参数适当增大,服务恢复正常;

火焰图性能分析 perf

2019-03-03T18:10:46+08:00

从 perf 命令（performance 的缩写）讲起，它是 Linux 系统原生提供的性能分析工具，会返回 CPU 正在执行的函数名以及调用栈（stack）。

通常，它的执行频率是 99Hz（每秒99次），如果99次都返回同一个函数名，那就说明 CPU 这一秒钟都在执行同一个函数，可能存在性能问题。

通过它，应用程序可以利用 PMU，tracepoint 和内核中的特殊计数器来进行性能统计。不但可以分析指定应用程序的性能问题，也可以用来分析内核的性能问题，当然也可以同时分析应用代码和内核，从而全面理解应用程序中的性能瓶颈

perf

yum install -y perf

perf top

用于实时显示当前系统的性能统计信息。该命令主要用来观察整个系统当前的状态，比如可以通过查看该命令的输出来查看当前系统最耗时的内核函数或某个用户进程

perf record

perf record -a -e cycles -o cycle.perf -g sleep 10

-a 对所有 CPU 采样 -o 输出文件名，如果不指定，默认生成 perf.data -g 额外记录函数调用关系 sleep 10 采样 10s

perf record -F 99 -p 13204 -g -- sleep 30

perf record表示记录 -F 99表示每秒99次 -p 13204是进程号，即对哪个进程进行分析 -g 表示记录调用栈 sleep 30 如上所述则是持续30秒。

perf record命令可以统计每个调用栈出现的百分比，然后从高到低排列。

perf report -n --stdio

perf report

perf report -i <file>

-i 指定 perf record 生成的 perf data 文件，如果不指定 -i 则默认分析 perf.data 文件。 –max-stack=0 只输出第一层函数调用 –stdio 输出到标准输出

生成火焰图

git clone https://github.com/brendangregg/FlameGraph.git

1、第一步

perf record -e cpu-clock -g -p 28591

Ctrl+c结束执行后，在当前目录下会生成采样数据perf.data.

2、第二步用perf script工具对perf.data进行解析

perf script -i perf.data &> perf.unfold

3、第三步

将perf.unfold中的符号进行折叠：

./stackcollapse-perf.pl perf.unfold &> perf.folded

4、最后生成svg图：

./flamegraph.pl perf.folded > perf.svg

使用管道将上面的流程简化为一条命令

perf script -i perf.data| ./stackcollapse-perf.pl | ./flamegraph.pl > process.svg

java 应用火焰图生成

jvm-profiling-tools

获取程序

wget https://github.com/jvm-profiling-tools/async-profiler/releases/download/v1.5/async-profiler-1.5-linux-x64.tar.gz

./profiler.sh -d 30 -f /tmp/elasticsearch.svg 48211

详细使用查看官方介绍即可
下图是我获取elasticsearch 进程的性能图例

参考链接

JVM 内存监控

2019-02-15T18:10:46+08:00

系统内存并不能反应JVM内存情况，经常碰到JVM内存满掉而系统内存大量空余的情况,或者是由于没有对JVM内存进行有效的监控导致无法及时获取应用最新使用状态；

jstat

gcutil

$JAVA_HOME/bin/jstat -gcutil VMID interval count

每隔1毫秒输出

$JAVA_HOME/bin/jstat -gcutil 3548 1

每隔1秒输出,且统计五次

$JAVA_HOME/bin/jstat -gcutil 3548 1000 5

➜  ~ $JAVA_HOME/bin/jstat -gcutil 3548
  S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT
  0.00 100.00  34.69  78.45  96.39  91.25 881491 30503.989     9  350.987 30854.976

取heap size 已使用量百分比值

➜  ~ $JAVA_HOME/bin/jstat -gcutil 3548|awk 'NR==2 {print $3}'
63.28

gccause

$JAVA_HOME/bin/jstat -gccause VMID interval count

gccapacity

$JAVA_HOME/bin/jstat -gccapacity VMID interval count

Jconsole

认证连接

JAVA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=21812 \
-Dcom.sun.management.jmxremote.authenticate=true \
-Dcom.sun.management.jmxremote.password.file=/etc/jmx/jmx-spark.password \
-Dcom.sun.management.jmxremote.access.file=/etc/jmx/jmx-spark.access \
-Dcom.sun.management.jmxremote.ssl=false"

密码文件示例： $JAVA_HOME/jre/lib/management

无认证连接

JAVA_OPTS="-Dcom.sun.management.jmxremote.port=8999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.manageme
nt.jmxremote.ssl=false"

客户端

Mac & Windowns 安装配置jdk 后执行jsconsole 即可

helm 私有仓库

2019-01-10T20:35:46+08:00

helm 私有本地仓库配置

关于helm 安装配置参考Kubernetes helm 包管理工具

开启服务

mkdir /opt/helm/charts
helm serve --address helm.jevic.cn:8879 --repo-path /opt/helm/charts &>/dev/null &

添加仓库:

helm repo add jevic http://hub.jevic.cn:8879

查看仓库列表

helm repo list

NAME     	URL
local    	http://127.0.0.1:8879
incubator	https://aliacs-app-catalog.oss-cn-hangzhou.aliyuncs.com/charts-incubator/
stable   	https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
aliyun   	https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts-incubator/
jevic    	http://helm.jevic.cn:8879

应用上传

## 打包应用
cp -r demoapp /opt/helm/charts
cd /opt/helm/charts
helm package demoapp --save=false
## 更新索引
cd /opt/helm/charts
helm repo index --url=http://helm.jevic.cn:8879 .
helm repo update

部署

## search
helm search demo

NAME         	CHART VERSION	APP VERSION	DESCRIPTION
jevic/demoapp	0.3.0        	1.0        	A Helm chart DemoApp for Kubernetes


## install
helm install -n appv3 jevic/demoapp

## get pods
kubectl get pods |grep appv3

appv3-demoapp-79b7f754d5-zbkvd                   1/1     Running   0          91s

## list 
helm ls 

grafana html 请求显示docker镜像信息

2018-11-27T19:56:06+08:00

通过grafana 请求应用接口显示Docker镜像的构建版本信息..

图例

api

curl -sL http://192.168.2.146:9010/version |python -mjson.tool
{
    "data": {
        "buildtime": "2018-11-27_17:25:43",
        "hash": "7cd3a541053a9d67eee1706d380af3fe",
        "version": "fa92b4d"
    },
    "ok": true
}

html 代码

<div class="version">

</div>

<style>
    .version ul{
        list-style:none;
        padding:0px;
        margin:0px;
        width:590px;
        height:20px;
        line-height:20px;
        font-size:12px;
    }
    .version ul li{
        display:block;
        width:33%;
        float:left;
        text-indent:2em
    }
    .version .th{
        background:"gray";
        font-weight:bold;
        border-top:0px
    }
</style>

<script>
    var hosts = ["192.168.2.2","192.168.2.27","192.168.2.23","192.168.2.25"];
    var port = 6010;

    function fetch(host,port) {
        $.ajax({
            type: "GET",
            url: "http://"+host+":"+port+"/version",
            dataType: "json",
            success: function(data){
                var id = "seg-"+host.replace(/\./g,"")+port;
                $("#"+id+">.version").text(data["data"]["version"])
                $("#"+id+">.buildtime").text(data["data"]["buildtime"])
            }
        });
    }

    function renderFrame(hosts, port) {
        var header = "<ul class=\"th\">\n" +
            "        <li>Host</li>\n" +
            "        <li>版本号</li>\n" +
            "        <li>构建时间</li>\n" +
            "    </ul>";
        var template = "<ul id=\"\">\n" +
            "        <li></li>\n" +
            "        <li class=\"version\"></li>\n" +
            "        <li class=\"buildtime\"></li>\n" +
            "    </ul>";

        var frame = header;
        for(var i = 0; i < hosts.length; i++){
            var host = hosts[i];
            var id = "seg-"+host.replace(/\./g,"")+port;
            var item = template.replace("",id).replace("",host+":"+port);
            frame += item
        }

        $(".version").append(frame);
    }

    $(document).ready(function () {
        renderFrame(hosts, port);

        for(var i = 0; i < hosts.length; i++){
            var host = hosts[i];
            window.setInterval(function (host) {
                fetch(host,port)
            },10000,host)
        }

    })
</script>

Rancher Dashboard 管理kubernetes集群

2018-11-25T20:35:46+08:00

相比kubernetes Dashboard Rancher管理面板简化了很多相对比较复杂的操作;

导入kubernetes 集群

这里直接给出截图操作示例

点击导入集群配置集群名称,Roles 默认即可,然后点击 Create

授权用户: 根据提示获取kubelet configuration file 中的用户名称

# cat ~/.kube/config|grep user ## /etc/kubernetes/kubelet.kubeconfig
    user: default-auth
users:
  user:
  
# kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user default-auth

部署rancher agent

需要注意的是,如果配置了SSL 认证,则最好使用第二条命令，否则可以使用第一条命令执行部署

此次使用第二条命令,以免出现证书不可用时忽略报错

curl --insecure -sfL https://192.168.2.218:8443/v3/import/wrmxhfxc7vcfznb9hssmqt9llx278ljqgfj49k5sxhtszrs68r6lmd.yaml | kubectl apply -f -

最后只需要等待agent 部署成功即可

Elastic Stack 6.4数据探索可视化 Demo

2018-11-23T19:56:06+08:00

基础安装配置请参考官网或者ELK 分类下其他教程示例

nginx

nginx.conf

指定日志格式如下:

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    log_format jsonTest '{"@timestamp":"$time_iso8601",'
                  '"host":"$server_addr",'
                  '"service":"nginxTest",'
                  '"trace":"$upstream_http_ctx_transaction_id",'
                  '"log":"log",'
                  '"clientip":"$remote_addr",'
                  '"remote_user":"$remote_user",'
                  '"request":"$request",'
                  '"http_user_agent":"$http_user_agent",'
                  '"size":$body_bytes_sent,'
                  '"responsetime":$request_time,'
                  '"upstreamtime":"$upstream_response_time",'
                  '"upstreamhost":"$upstream_addr",'
                  '"http_host":"$host",'
                  '"url":"$uri",'
                  '"domain":"$host",'
                  '"xff":"$http_x_forwarded_for",'
                  '"referer":"$http_referer",'
                  '"status":"$status"}';

    access_log  /var/log/nginx/access.log  jsonTest;

elasticsearch

模板配置

template01

没有配置mapping 不指定字段类型

{
  "order": 0,
  "index_patterns": [
    "nginxlog-*"
  ],
  "settings": {
    "index": {
      "refresh_interval": "60s",
      "number_of_shards": "1",
      "auto_expand_replicas": "0-1",
      "number_of_replicas": "1"
    }
  },
  "mappings": {},
  "aliases": {}
}

template02

配置mapping 指定字段类型

{
  "order": 0,
  "index_patterns": [
    "testnginxlog-*"
  ],
  "settings": {
    "index": {
      "number_of_shards": "1",
      "auto_expand_replicas": "0-1",
      "refresh_interval": "60s"
    }
  },
  "mappings": {
    "doc": {
    "dynamic":true,
      "properties": {
        "clientip": {
          "type": "ip"
        },
        "host": {
          "type": "ip"
        },
        "responsetime": {
          "type": "float"
        },
        "size": {
          "type": "long"
        },
        "status": {
          "type": "long"
        },
        "upstreamtime": {
          "type": "float",
          "index": false
        }
      }
    }
  },
  "aliases": {}
}

通过dynamic 参数来控制字段的新增
- true(默认) 允许自动新增字段
- false 不允许新增字段,但文档可以正常写入,但无法对字段进行查询等操作
- strict 文档不能写入,报错
index
- 控制当前字段是否索引,默认为true,记录索引,false不记录,不可搜索
- 当索引的某些字段不需要被查询搜索时可标记index 为false,以此来减少存储空间(没有倒排索引)

logstash

示例中给出了两个索引名称,分别为:
- nginxlog_*
- testnginxlog-*
创建两个配置文件,修改对应的索引名称即可

input {
   file {
    type => "proxys"
    path => ["/var/log/nginx/access.log"]
    add_field => { "sip" => "192.168.2.243" }
    ## 是否从头读取数据这里注释掉只读取最新数据
#    start_position => "beginning"
#    sincedb_path => "/dev/null"
#    ignore_older => "99999999999"
    codec => "json"
  }
}

filter {
date {
        match => ["timestamp","UNIX"]
        target => "@timestamp"
 }
#mutate {
#        remove_field => ["path","host","tags","message"]
#}
}

output{
        elasticsearch {
        hosts => ["http://192.168.2.221:9200"]
        #index => "nginxlog_%{+YYYY.MM.dd}"
        index => "testnginxlog-%{+YYYY.MM.dd}"
    }
 #      stdout {
 #           codec => rubydebug
 #      }
}

启动进程

$LOGSTAH_PATH/bin/logstash -f nginxlog.conf --log.level=info --path.data=/tmp/nginxlog

$LOGSTAH_PATH/bin/logstash -f testnginxlog.conf --log.level=info --path.data=/tmp/testnginxlog

查看索引mapping

在 elasticsearch 2.x 版本，字符串数据只有string类型更新到5.x版本后，取消了string 数据类型，代替它的是 keyword 和 text 数据类型

Mapping 类似数据库中的表结构定义,主要作用如下:
- 定义index 下的字段名(Field Name)
- 定义字段的类型,比如: 数值型、字符串型、布尔型等
- 定义倒排索引相关配置,比如是否索引、记录position等

默认: “key”:”value” 格式的都默认为text类型而对于”key”: value 格式的会根据数值类型进行格式匹配

例如:

{
....
"responsetime": 0.001, 
"upstreamtime": "0.001",
...
}

responsetime: 匹配为fload 类型
upstreamtime: 匹配为text类型,此时如果想要做聚合查询以及计算数值需要配置mapping 指定字段类型
1. text类型：会分词，先把对象进行分词处理，然后再再存入到es中。

当使用多个单词进行查询的时候，当然查不到已经分词过的内容！

1. keyword：不分词，没有把es中的对象进行分词处理，而是存入了整个对象！

这时候当然可以进行完整地查询！默认是256个字符！

“ignore_above”: 256 详情查看官网文档

nginxlog_*

testnginxlog-*

kibana

添加索引

监控图表

grafana

runing of docker

默认用户名密码: admin:admin

docker run -d \
--name grafana \
-p 3000:3000 \
-v /var/lib/grafana:/var/lib/grafana
-v /etc/grafana:/etc/grafana
-v /etc/localtime:/etc/localtime
grafana/grafana

数据源

这里ES 版本选择5.6+即可

demo

需要注意的是: 对于nginxlog_* 这个索引. Group by 时写入的是 clientip.keyword; 因为没有指定字段类型,如果直接写入 clientip 则会报错无法显示;

变量

配置完成后,点击add 添加即可

变量引用,和shell 变量调用方式一样使用 ${变量名}即可

clientip: $ip

注意: 这里Group by 时状态码字段直接指定为: status 即可因为根据前面我们的配置,对testnginxlog-* 索引模板中指定了字段类型

图表展示:

elastic-stack 6.5

2018-11-22T19:56:06+08:00

在以往的旧版本(2.x,5.x) 每个索引可以存储不同类型的文档,

类比MySQL

index == database

_type == table

6.x 版本开始移除了_type 也就是每个索引只有一种类型！！！

x-pack 从6.3版本开始已经内置在elasticsearch,kibana 当中无需另行安装!

elasticsearch

官网下载
下载解压

配置基础环境

# cat /etc/security/limit.conf
* soft nofile 65536
* hard nofile 65536
* soft nproc unlimited
* hard nproc unlimited
es soft memlock unlimited
es hard memlock unlimited
  
# cat /etc/sysctl.conf
vm.swappiness = 1
vm.max_map_count=262144

添加 es 用户并授权
启动
授权 license （30天试用版）
- 证书申请: https://register.elastic.co/marvel_register

curl -H "Content-Type:application/json" -XPOST  http://192.168.2.221:9200/_xpack/license/start_trial?acknowledge=true

x-pack 开启认证

配置用户名密码:
- $ES_PATH:/bin/elasticsearch-setup-passwords interactive
- 根据提示一步步设置密码即可

修改 elasticsearch 配置文件

# tail elasticsearch.yml -n 1
xpack.security.enabled: true  ## 开启认证

重启ES，再次访问则需要输入用户名密码
修改密码

curl -H "Content-Type:application/json" -XPOST -u elastic 'http://192.168.2.221:9200/_xpack/security/user/elastic/_password' -d '{ "password" : "123456" }'

kibana

配置对应的用户名密码以及 ES 链接地址
配置文件添加此配置:
- xpack.security.enabled: true
启动即可

cerebro

github
具体步骤查看文档即可

sql 插件

github

转载请注明出处，本文采用 CC4.0 协议授权

kubernetes nginx-ingress and traefik

2018-10-25T22:35:46+08:00

在Kubernetes中，服务和Pod的IP地址仅可以在集群网络内部使用，对于集群外的应用是不可见的。为了使外部的应用能够访问集群内的服务，在Kubernetes中可以通过NodePort和LoadBalancer这两种类型的服务，或者使用Ingress。Ingress本质是通过http代理服务器将外部的http请求转发到集群内部的后端服务。

一. 证书(可选配置)

使用已购买的证书
配置自签名证书

# openssl genrsa -out jevic_key.pem 4096
Generating RSA private key, 4096 bit long modulus
.....................++
.....++
e is 65537 (0x10001)
[root@master219 ssl]# openssl req -new -x509 -key jevic_key.pem -out root.crt -days 3650
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:GD
State or Province Name (full name) []:GD
Locality Name (eg, city) [Default City]:SZ
Organization Name (eg, company) [Default Company Ltd]:jevic
Organizational Unit Name (eg, section) []:jevic
Common Name (eg, your name or your server's hostname) []:jevic.cn
Email Address []:admin@jevic.cn
[root@master219 ssl]# ls
jevic_key.pem  root.crt

二. nginx-ingress

2.1 部署

如果kubernetes 集群启用了RBAC，则一定要加rbac.create=true参数 helm 安装请参考Kubernetes helm 包管理工具

helm install stable/nginx-ingress --set controller.hostNetwork=true，rbac.create=true

查看状态:

# helm list
NAME            	REVISION	UPDATED                 	STATUS  	CHART              	APP VERSION	NAMESPACE
quiet-abalone   	1       	Thu Nov 29 19:55:28 2018	DEPLOYED	nginx-ingress-0.9.5	0.10.2     	default

# kubectl get all --all-namespaces -l app=nginx-ingress
NAMESPACE   NAME                                                               READY     STATUS    RESTARTS   AGE
default     pod/quiet-abalone-nginx-ingress-controller-7b8b745f96-gczjl        1/1       Running   1          5d
default     pod/quiet-abalone-nginx-ingress-default-backend-54f4fb569c-ndh4x   1/1       Running   1          5d

NAMESPACE   NAME                                                  TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
default     service/quiet-abalone-nginx-ingress-controller        LoadBalancer   10.254.235.53   <pending>     80:45608/TCP,443:40984/TCP   5d
default     service/quiet-abalone-nginx-ingress-default-backend   ClusterIP      10.254.20.221   <none>        80/TCP                       5d

NAMESPACE   NAME                                                          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
default     deployment.apps/quiet-abalone-nginx-ingress-controller        1         1         1            1           5d
default     deployment.apps/quiet-abalone-nginx-ingress-default-backend   1         1         1            1           5d

NAMESPACE   NAME                                                                     DESIRED   CURRENT   READY     AGE
default     replicaset.apps/quiet-abalone-nginx-ingress-controller-7b8b745f96        1         1         1         5d
default     replicaset.apps/quiet-abalone-nginx-ingress-default-backend-54f4fb569c   1         1         1         5d

2.2 nginx 负载均衡测试

2.2.1 nginx-demo.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-demo
  labels:
    app: my-nginx
spec:
  type: NodePort
  ports:
  - port: 80
    protocol: TCP
    name: http
  selector:
    app: my-nginx
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: my-nginx
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: my-nginx
    spec:
      containers:
      - image: nginx/nginx:1.7.9
        name: nginx-demo
        ports:
        - containerPort: 80

2.2.2 ingress-nginx-demo.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress-demo
  namespace: default
#  annotations:
#    kubernetes.io/ingress.class: "public"
#    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: test.nginx.com
    http:
      paths:
      - backend:
          serviceName: nginx-demo
          servicePort: 80
        path: /

三. traefik

官网 yaml文档 docs.traefik.io

3.1 traefik-insecure.yaml

只开启了http 请求访问

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: traefik-conf
  namespace: kube-system
data:
  traefik.toml: |
    insecureSkipVerify = true
    defaultEntryPoints = ["http"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      volumes:
      - name: config
        configMap:
          name: traefik-conf
      containers:
      - image: k8s.jevic.cn/kubernetes/traefik:1.7.3
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: admin
          containerPort: 8080
        securityContext:
          privileged: true
        args:
        - --configfile=/config/traefik.toml
        - -d
        - --web
        - --kubernetes
        - --web.address=0.0.0.0:8081
        volumeMounts:
        - mountPath: "/config"
          name: "config"
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 8080
      name: admin
    - protocol: TCP
      port: 443
      name: https
  type: NodePort
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: ingress.jevic.cn
    http:
      paths:
      - backend:
          serviceName: traefik-web-ui
          servicePort: 80

kubectl create -f traefik-insecure.yaml

然后等待服务正常启动即可

# kubectl get pod -n kube-system -l k8s-app=traefik-ingress-lb
NAME                               READY     STATUS    RESTARTS   AGE
traefik-ingress-controller-mv8cg   1/1       Running   0          4d
traefik-ingress-controller-vvfc7   1/1       Running   0          4d

3.2 traefik-https.yaml

开启http 和 https 访问

3.2.1 创建 secret

这里使用购买的证书,也可以使用自签名证书测试；

kubectl create secret tls ssl --cert=jevic.cert --key=jevic.key -n kube-system
kubectl get secret -n kube-system

这里注意namespace 一定要配置为kube-system

3.2.2 traefik-https.yaml

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: traefik-conf
  namespace: kube-system
data:
  traefik.toml: |
    insecureSkipVerify = true
    defaultEntryPoints = ["http","https"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      [entryPoints.https]
      address = ":443"
        [entryPoints.https.tls]
          [[entryPoints.https.tls.certificates]]
          CertFile = "/ssl/tls.crt"
          KeyFile = "/ssl/tls.key"
---
kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      volumes:
      - name: ssl
        secret:
          secretName: ssl
      - name: config
        configMap:
          name: traefik-conf
      containers:
      - image: k8s.jevic.cn/kubernetes/traefik:latest
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: admin
          containerPort: 8081
        securityContext:
          privileged: true
        args:
        - --configfile=/config/traefik.toml
        - -d
        - --web
        - --kubernetes
        - --web.address=0.0.0.0:8081
        volumeMounts:
        - mountPath: "/ssl"
          name: "ssl"
        - mountPath: "/config"
          name: "config"
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 8081
      name: admin
    - protocol: TCP
      port: 443
      name: https
  type: NodePort
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - port: 80
    targetPort: 8081
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: ingress.jevic.cn
    http:
      paths:
      - backend:
          serviceName: traefik-web-ui
          servicePort: 80

默认端口: 8080 这里手动指定了web-ui端口为:8081

- --web.address=0.0.0.0:8081

3.3 目录结构

├── jevic.cert
├── jevic.key
├── nginx-demo
│   ├── ingress-nginx-traefik.yaml
│   └── nginx-demo.yaml
├── traefik-https.yaml
└── traefik-insecure.yaml

3.4 traefik nginx-demo

nginx-deployment-demo.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-demo
  labels:
    nginx: nginx-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      nginx: nginx-demo
  template:
    metadata:
      labels:
        nginx: nginx-demo
    spec:
      containers:
      - name: nginx-demo
        image: nginx/nginx:1.7.9
        ports:
        - name: https
          containerPort: 443
        - name: http
          containerPort: 80

nginx-svc-demo.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx-demo
  labels:
    nginx: nginx-demo
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
    name: http
  - port: 443
    protocol: TCP
    targetPort: 443
    name: https
  selector:
    nginx: nginx-demo

ingress-nginx-traefik.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress-demo
  namespace: default
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: nginx.jevic.cn
    http:
      paths:
      - backend:
          serviceName: nginx-demo
          servicePort: 80
        path: /

部署

# ls
ingress-nginx-traefik.yaml  nginx-deployment-demo.yaml  nginx-rc-demo.yaml  nginx-svc-demo.yaml
# kubectl apply -f nginx-deployment-demo.yaml
deployment.apps "nginx-demo" created
# kubectl create -f nginx-svc-demo.yaml
service "nginx-demo" created
# kubectl create -f ingress-nginx-traefik.yaml
ingress.extensions "nginx-ingress-demo" created
# kubectl get all -l nginx=nginx-demo
NAMESPACE   NAME                            READY     STATUS    RESTARTS   AGE
default     pod/nginx-demo-c8765675-phr2w   1/1       Running   0          1m
default     pod/nginx-demo-c8765675-z7tgn   1/1       Running   0          1m

NAMESPACE   NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
default     deployment.apps/nginx-demo   2         2         2            2           1m

NAMESPACE   NAME                                  DESIRED   CURRENT   READY     AGE
default     replicaset.apps/nginx-demo-c8765675   2         2         2         1m

kafka 压力测试

2018-10-21T18:10:46+08:00

kafka 内置提供了压测脚本

kafka 版本

$ find /opt/kafka/libs/ -name \*kafka_\* | head -1 | grep -o '\kafka[^\n]*'
kafka/libs/kafka_2.11-1.0.0.2.6.5.1050-37-javadoc.jar

Producer测试

$ ./bin/kafka-producer-perf-test.sh \
--topic test \
--num-records 1000000 \
--record-size 3000 \
--throughput 20000 \
--producer-props bootstrap.servers=192.168.25.198:6667,192.168.25.199:6667,192.168.25.200:6667,192.168.25.201:6667,192.168.25.202:6667

99914 records sent, 19978.8 records/sec (57.16 MB/sec), 124.9 ms avg latency, 566.0 max latency.
99292 records sent, 19858.4 records/sec (56.82 MB/sec), 7.1 ms avg latency, 214.0 max latency.
96498 records sent, 19299.6 records/sec (55.22 MB/sec), 117.3 ms avg latency, 1096.0 max latency.
95601 records sent, 19120.2 records/sec (54.70 MB/sec), 269.1 ms avg latency, 2194.0 max latency.
108777 records sent, 21755.4 records/sec (62.24 MB/sec), 340.9 ms avg latency, 2665.0 max latency.
100020 records sent, 20004.0 records/sec (57.23 MB/sec), 1.3 ms avg latency, 20.0 max latency.
100051 records sent, 20010.2 records/sec (57.25 MB/sec), 1.3 ms avg latency, 20.0 max latency.
99987 records sent, 19997.4 records/sec (57.21 MB/sec), 1.3 ms avg latency, 27.0 max latency.
100036 records sent, 20007.2 records/sec (57.24 MB/sec), 1.3 ms avg latency, 21.0 max latency.
1000000 records sent, 19992.402887 records/sec (57.20 MB/sec), 87.97 ms avg latency, 2665.00 ms max latency, 1 ms 50th, 500 ms 95th, 2202 ms 99th, 2613 ms 99.9th.

–topic topic名称，本例为test_perf

–num-records 总共需要发送的消息数，本例为1000000

–record-size 每个记录的字节数，本例为1000

–throughput 每秒钟发送的记录数，本例为20000

–producer-props bootstrap.servers=localhost:9092 发送端的配置信息

上面示例可以看出平均每秒写入57.20MB数据,大概是19992条消息

每次写入的平均延迟是: 87.98 毫秒

最大延迟是: 2665.00 毫秒

Consumer 测试

./bin/kafka-consumer-perf-test.sh 
--topic test \
--fetch-size 1048576 \
--messages 1000000 \
--threads 20 \
--hide-header \
--broker-list 192.168.25.198:6667,192.168.25.199:6667,192.168.25.200:6667,192.168.25.201:6667,192.168.25.202:6667

–topic 指定topic的名称，本例为test_perf

–fetch-size 指定每次fetch的数据的大小，本例为1048576，也就是1M

–messages 总共要消费的消息个数，本例为1000000，100w

删除topic

./kafka-topics.sh --zookeeper 192.168.25.198:2181 --delete --topic test

可逐一增大数量级以及消息个数等参数对比结果;
详细的参数信息直接执行脚本查看帮助信息;