Implementing a Dynamic IP Blacklist with Nginx, Lua, and Redis

Requirement : To block certain crawlers or malicious users by creating a dynamic IP blacklist that can reject requests from listed IPs and set expiration times.

Environment preparation : Linux (CentOS7/Ubuntu), Redis 5.0.5, Nginx OpenResty.

Design options : OS‑level iptables rules – simple but manual and inflexible. Nginx deny option or Lua plugin – dynamic, supports distributed blocking via Redis but requires Lua knowledge. Application‑level check – easy to code but may affect performance under high concurrency. The chosen solution combines Nginx, Lua, and Redis for a lightweight, dynamic blacklist.

nginx.conf configuration

location / {
    # optional static file check (commented out)
    #if ($request_uri ~ .*\.(html|htm|jpg|js|css)) {
    #    access_by_lua_file /usr/local/lua/access_limit.lua;
    #}
    access_by_lua_file /usr/local/lua/access_limit.lua;  # apply Lua rate‑limit rules
    alias /usr/local/web/;
    index index.html index.htm;
}

Lua script (access_limit.lua)

/usr/local/lua/access_limit.lua
-- configuration parameters
local pool_max_idle_time = 10000          -- connection pool idle timeout (ms)
local pool_size = 100                     -- connection pool size
local redis_connection_timeout = 100     -- Redis connection timeout (ms)
local redis_host = "your redis host ip"
local redis_port = "your redis port"
local redis_auth = "your redis authpassword"
local ip_block_time = 120                -- block duration (seconds)
local ip_time_out = 1                    -- time window for counting (seconds)
local ip_max_count = 3                   -- max requests in the window

local function errlog(msg, ex)
    ngx.log(ngx.ERR, msg, ex)
end

local function close_redis(red)
    if not red then return end
    local ok, err = red:set_keepalive(pool_max_idle_time, pool_size)
    if not ok then
        ngx.say("redis connct err:", err)
        return red:close()
    end
end

local redis = require "resty.redis"
local client = redis:new()
local ok, err = client:connect(redis_host, redis_port)
if not ok then
    close_redis(client)
    ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
end
client:set_timeout(redis_connection_timeout)

local connCount, err = client:get_reused_times()
if 0 == connCount then
    local ok, err = client:auth(redis_auth)
    if not ok then
        errlog("failed to auth: ", err)
        return
    end
elseif err then
    errlog("failed to get reused times: ", err)
    return
end

local function getIp()
    local clientIP = ngx.req.get_headers()["X-Real-IP"]
    if clientIP == nil then
        clientIP = ngx.req.get_headers()["x_forwarded_for"]
    end
    if clientIP == nil then
        clientIP = ngx.var.remote_addr
    end
    return clientIP
end

local cliendIp = getIp()
local incrKey = "limit:count:" .. cliendIp
local blockKey = "limit:block:" .. cliendIp

local is_block, err = client:get(blockKey)
if tonumber(is_block) == 1 then
    ngx.exit(ngx.HTTP_FORBIDDEN)
    close_redis(client)
end

local ip_count, err = client:incr(incrKey)
if tonumber(ip_count) == 1 then
    client:expire(incrKey, ip_time_out)
end
if tonumber(ip_count) > tonumber(ip_max_count) then
    client:set(blockKey, 1)
    client:expire(blockKey, ip_block_time)
end

close_redis(client)

Summary of advantages :

Simple, lightweight configuration with minimal performance impact.

Multiple servers can share the blacklist via a common Redis instance.

Dynamic updates can be performed manually or through automation.

Extensions

Application scenarios include preventing malicious access, blocking crawlers, mitigating DDoS attacks, and limiting request frequency.

Advanced features may add anomaly detection with automatic banning, whitelist mechanisms, captcha verification for suspicious IPs, and statistical analysis of blacklist data for continuous improvement.