数据库 id 字段溢出

某个系统,突然数据库无法插入数据,报错如下

[2017-04-22 23:17:55] - [ERROR] - [utils.api.api:146]  - integer out of range
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
psycopg2.DataError: integer out of range

然后表结构是这样的

user=# select * from acl_ip_data_id_seq;
   sequence_name    | last_value | start_value | increment_by |      max_value      | min_value | cache_value | log_cnt | is_cycled | is_called 
--------------------+------------+-------------+--------------+---------------------+-----------+-------------+---------+-----------+-----------
 acl_ip_data_id_seq | 2147550630 |           1 |            1 | 9223372036854775807 |         1 |           1 |       7 | f         | t
(1 row)

看到21亿这个数字很熟悉,是 int 最大值,怀疑是数据库 id 字段达到了最大值,无法继续增长了。

这个表是 Django ORM 生成的,看了下 migration,这个字段是 AutoField,发现其实就是 IntegerField

https://docs.djangoproject.com/en/1.9/ref/models/fields/#autofield

但是只有 Django 1.10 才支持更大的整形 ID 字段

https://docs.djangoproject.com/en/1.10/ref/models/fields/#bigautofield

前几天刚看了饿了么的技术一个故障分析,是一样的原因 http://efs.ele.me/?p=246 结果过了几天就遇见了。

localhost 解析出 IPV6 的地址

入口是 Nginx,反代 uwsgi,日志中时不时出现下面的内容

[error] 54#54: *1644 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.5, server: _, request: "POST /log HTTP/1.1", upstream: "http://[::1]:10000/log", host: "mgt-api"

比较确定的是不是后面的 uwsgi 挂掉了,之后 tcpdump 抓包,发现每次出现问题的都是一个 IPV6 的地址,结合上面的日志恍然大悟,原来是 Nginx 反向代理写的是 localhost 而不是 127.0.0.1,而且在 host 文件中是

127.0.0.1	localhost
::1             localhos

gethostbyname 在多个 IP 的情况下,返回结果是随机的,导致有时候被解析到了 IPV6 的地址上了。

Django在 https 下增强的 CSRF 防护

某 Django 系统在 POST 的时候一直提示 CSRF 验证失败,即使看到的 CSRFToken 是没问题的,后来打开 DEBUG 看到提示,原因摘抄 Django 源码。

if request.is_secure():
    # Suppose user visits http://example.com/
    # An active network attacker (man-in-the-middle, MITM) sends a
    # POST form that targets https://example.com/detonate-bomb/ and
    # submits it via JavaScript.
    #
    # The attacker will need to provide a CSRF cookie and token, but
    # that's no problem for a MITM and the session-independent
    # nonce we're using. So the MITM can circumvent the CSRF
    # protection. This is true for any HTTP connection, but anyone
    # using HTTPS expects better! For this reason, for
    # https://example.com/ we need additional protection that treats
    # http://example.com/ as completely untrusted. Under HTTPS,
    # Barth et al. found that the Referer header is missing for
    # same-domain requests in only about 0.2% of cases or less, so
    # we can use strict Referer checking.
    referer = force_text(
        request.META.get('HTTP_REFERER'),
        strings_only=True,
        errors='replace'
    )
    if referer is None:
        return self._reject(request, REASON_NO_REFERER)

    referer = urlparse(referer)

    # Make sure we have a valid URL for Referer.
    if '' in (referer.scheme, referer.netloc):
        return self._reject(request, REASON_MALFORMED_REFERER)

    # Ensure that our Referer is also secure.
    if referer.scheme != 'https':
        return self._reject(request, REASON_INSECURE_REFERER)

    # If there isn't a CSRF_COOKIE_DOMAIN, assume we need an exact
    # match on host:port. If not, obey the cookie rules.
    if settings.CSRF_COOKIE_DOMAIN is None:
        # request.get_host() includes the port.
        good_referer = request.get_host()
    else:
        good_referer = settings.CSRF_COOKIE_DOMAIN
        server_port = request.get_port()
        if server_port not in ('443', '80'):
            good_referer = '%s:%s' % (good_referer, server_port)

    # Here we generate a list of all acceptable HTTP referers,
    # including the current host since that has been validated
    # upstream.
    good_hosts = list(settings.CSRF_TRUSTED_ORIGINS)
    good_hosts.append(good_referer)

    if not any(is_same_domain(referer.netloc, host) for host in good_hosts):
        reason = REASON_BAD_REFERER % referer.geturl()
        return self._reject(request, reason)