需求: 根据不同IP返回不同后端，实现一个简单版的一个灰度发布功能

使用到的模块

该模块已内置于OpenResty可以直接使用

方法一

OpenResty配置

  1. 先定义二个upstream

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

    # 定义一个default,为默认后端
upstream default {
    server 127.0.0.1:8080 weight=10;
    server 127.0.0.1:8081 weight=10;
}

# 定义一个灰度,匹配的IP由灰度提供服务
upstream stage {
    server 127.0.0.1:8090 weight=10;
    server 127.0.0.1:8091 weight=10;
}

  2. 通过rewrite_by_lua_file指令来实现不同后端

     1
 2
 3
 4
 5
 6
 7
 8
 9
10

    server {
    ...
    location / {
        ...
        set $backend "default";
        rewrite_by_lua_file conf/lua/set_upstream.lua;
        proxy_pass http://$backend;
    }
    ...
}

  3. set_upstream.lua

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

    #!/usr/bin/lua

local redis_host = "127.0.0.1"
local redis_port = 6379
-- key 为Redis的Set类型
local key = "td:stage:shops"


local function close_redis(red)
    if not red then
        return
    end
    local pool_max_idle_time = 100
    local pool_size = 20
    local ok, err = red:set_keepalive(pool_max_idle_time, pool_size)
    if not ok then
        ngx.log(ngx.ERR, "set keepalive err ", err)
    end
end


local function get_ip()
    local client_ip = ngx.req.get_headers()["X-Real-IP"]
    if client_ip == nil then
        client_ip = ngx.req.get_headers()["x_forwarded_for"]
    end

    if client_ip == nil then
        client_ip = ngx.var.remote_addr
    end

    return client_ip
end


local function is_stage(ip)
    local result = 0
    local resty_redis = require "resty.redis"
    local redis = resty_redis:new()
    local ok, err = redis:connect(redis_host, redis_port)
    if err then
        colse_redis(redis)
        ngx.log(ngx.ERR, "Connect to redis failed ", err)
    else
        -- ngx.log(ngx.ERR, "Connect success")
        local _result, err = redis:sismember(key, ip)
        if err then
            ngx.log(ngx.ERR, "Get key failed", err)
        else
            result = _result
        end
    end
   
    if result == 1 then
        return true
    else
        return false
    end
end


local ip = get_ip()
if is_stage(ip) then
    ngx.var.backend = "stage"
else
    ngx.var.backend = "default"
end

方法二

前面的方法虽然是实现了，但总感觉不够优雅。而且做完压力测试后发现，以100的并发来压测，虽然增加的响应时间忽略不计，但对网络IO增长还是蛮多，连接Redis后的TIME_WAIT状态达到9000多，想想也是有点恐怖。

在此方法中，优化连接Redis的连接，因为IO都集中在频繁连接/断开Redis。通过lua_shared_dict，init_by_lua_file与set_by_lua_file实现

1. 定义二个upstream

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

...
upstream default {
    server localhost:80 weight=10;
    server localhost:82 weight=20;
}

upstream stage {
    server localhost:90 weight=10;
    server localhost:92 weight=10;
}

2. 通过set_by_lua_file指令，动态提供后端upstream

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

...
http {
    ...
    # 共享内存区域，用于所有的work进程变量共享
    lua_shared_dict ips 100m;
    init_by_lua_file conf/lua/init.lua;
    ...
    server {
        ...
        location / {
            ...
            # 设置upstream,后面是参数，第一个是默认的upstream名称，第二个是stage的名称。
            # 要同定义的upstream相对应起来,这样可以复用Lua文件
            set_by_lua_file $backend conf/lua/set_upstream_by_ip.lua default stage ips;
            proxy_pass http://$backend;
        }
        ...
    }
    ...
}
...

3. init.lua

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#!/usr/bin/lua

local host = "127.0.0.1"
local port = 6380
local key = "td:stage:shops"
local ngx_dict = "ips"


local function init_dict(redis_key, ngx_dict)
    local cmd = string.format("echo 'smembers %s' | redis-cli -h %s -p %d", redis_key, host, port)
    local f = io.popen(cmd)
    local dict = ngx.shared[ngx_dict]
    local line = f:read()
    dict:flush_all()
    while line then
        dict:set(line, true)
        -- ngx.log(ngx.ERR, "line is ", line)
        line = f:read()
    end
    f:close()
end

init_dict(key, ngx_dict)

4. set_upstream_by_ip.lua

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

local function get_ip()
    local client_ip = ngx.req.get_headers()["X-Real-IP"]
    if client_ip == nil then
        client_ip = ngx.req.get_headers()["x_forwarded_for"]
    end

    if client_ip == nil then
        client_ip = ngx.var.remote_addr
    end

    return client_ip
end

local default, stage, ngx_dict_name = ...
local remote_ip = get_ip()
local ngx_dict = ngx.shared[ngx_dict_name]
local result = ngx_dict:get(remote_ip)
if result then
    return stage
else
    return default
end

方法二的缺点就是不能实时更新，因为只有在init阶段写入共享内存，因此当要更新IP时，只能Reload.解决方法就是每分钟重载OpenResty.以下配合脚本，只有当Redis中的Key有变动时才更新

1

*/1 * * * * /root/reload.sh

reload.sh

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

#!/bin/bash
export  PATH
set -e

host="localhost"
pass="pass"
port=6380
id_key="td:stage:shops"
ip_key="td:stage:ips"

touch keys.txt

old_keys=$(cat keys.txt)
id_keys=$(echo "smembers ${id_key}" | redis-cli -h ${host} -p ${port} -a ${pass} -n 0)
ip_keys=$(echo "smembers ${ip_key}" | redis-cli -h ${host} -p ${port} -a ${pass} -n 0)
keys=$(echo ${id_keys}${ip_keys})

if [ "${keys}"x = "${old_keys}"x ];then
  echo "$(date) Key no change,Nothing to do" >> reload.log
else
  echo "$(date) key changed,Reoload" >> reload.log
  openresty -t && systemctl reload openresty
  echo ${keys} > keys.txt
fi

测试

往Redis中设置Key

1

echo "sadd td:stage:shops 192.168.0.100" | redis-cli -x

通过启用log_format中的request_time时间对比,使用了rewrite_by_lua_file与未使用相比,时间上没差距.既使Redis挂了,也不会产生阻塞导致响应时间过长,而是会直接使用默认("default")的后端

其它说明

测试发现有一个坑，只能用于整个路径全部代理，如下

1
2
3
4
5

...
location / {
    ...
    proxy_pass http://$backend;
}

下面的加路径代理的就会有问题:

1
2
3
4

...
location /path/ {
    proxy_pass http://$backend/;
}

可能是并不能很好识别最后的"/所致