初学Django服务端

作为一个iOS初学Django服务端，并为多个小程序提供稳定的服务

python的web框架中有Django和Flask，选择Django的原因就是生态完善，工具齐全。缺点就是臃肿、沉重。
Flask是微框架，很多东西都是插件化，如果不熟悉服务端开发的话上手起来并不容易，很多东西都要自己造轮子，定制化强。

所以最终选择python==3.6.8和Django==2.1.10

Django

REST

个人觉得是个规范，可以根据业务需求具体遵守。

Environment

环境区分，使用环境变量控制："DJANGO_SETTINGS_MODULE": "project_name.settings_debug"

1. settings: 基础配置
2. settings_debug：本地调试配置
3. settings_test：测试环境配置
4. settings_master：正式环境配置

Mysql

• mysqlclient：Django默认使用

使用了本地的mysqlclient的头文件，由于需要使用utf8mb4（5.5.3开始支持），而当前部署的服务器的mysql-client版本过低（5.1.73），所以无法使用。

• PyMySQL：由python重写，但是从Django2.2开始不支持

utf8mb4无法设置

# 当本地mysql-client版本过低时，使用'charset': 'utf8mb4'无效时
# 需要加上'init_command': "set names utf8mb4",
'default': {
   'NAME': 'NAME',
   'ENGINE': 'django.db.backends.mysql',
   'HOST': 'HOST',
   'PORT': 'PORT',
   'USER': 'USER',
   'PASSWORD': 'PASSWORD',
   'OPTIONS': {
       'charset': 'utf8mb4',
       'init_command': "set names utf8mb4",
   },
},

# requirements.txt 
# Do not upgrade Django unless https://github.com/PyMySQL/PyMySQL/issues/790 has been resolved.
Django==2.1.10
# mysqlclient==1.4.2 由于服务器mysql连接器版本过低（部分头文件没有）
PyMySQL==0.9.3

自定义数据库路由

线上环境不允许ORM建表，所以Django的Admin表从测试环境读取
由此可以实现不同的applabel读取不同的数据库

# settings.py
DATABASE_APPS_MAPPING = {
    'applabel0': 'default',
    'applabel1': 'master',
}
# 自定义数据库路由
DATABASE_ROUTERS = [
    'path.db_router.DatabaseAppsRouter'
]


# db_router.py
from django.conf import settings

DATABASE_MAPPING = settings.DATABASE_APPS_MAPPING


class DatabaseAppsRouter(object):
    """
    A router to control all database operations on models for different
    databases.

    In case an app is not set in settings.DATABASE_APPS_MAPPING, the router
    will fallback to the `default` database.

    Settings example:

    DATABASE_APPS_MAPPING = {'app1': 'db1', 'app2': 'db2'}
    """

    def db_for_read(self, model, **hints):
        """"Point all read operations to the specific database."""
        # print('🍏', model._meta.app_label, 'read')
        if model._meta.app_label in DATABASE_MAPPING:
            return DATABASE_MAPPING[model._meta.app_label]
        return 'default'

    def db_for_write(self, model, **hints):
        """Point all write operations to the specific database."""
        # print('🍏', model._meta.app_label, 'write')
        if model._meta.app_label in DATABASE_MAPPING:
            return DATABASE_MAPPING[model._meta.app_label]
        return 'default'

    def allow_relation(self, obj1, obj2, **hints):
        """Allow any relation between apps that use the same database."""
        # print('🍏', obj1._meta.app_label, obj2._meta.app_label, 'relation')
        db_obj1 = DATABASE_MAPPING.get(obj1._meta.app_label)
        db_obj2 = DATABASE_MAPPING.get(obj2._meta.app_label)
        if db_obj1 and db_obj2:
            if db_obj1 == db_obj2:
                return True
            else:
                return False
        return True

    def db_for_migrate(self, db, app_label, model_name=None, **hints):
        # print('🍏', app_label, 'migrate')
        if db in DATABASE_MAPPING.values():
            return DATABASE_MAPPING.get(app_label) == db
        elif app_label in DATABASE_MAPPING:
            return False
        return True

Logging

日志记录

# 自定义日志记录样式
class EmojiLogFormatter(logging.Formatter):

    def formatMessage(self, record):
        emoji = ''
        if record.levelno == logging.INFO:
            emoji = '🍺'
        elif record.levelno == logging.WARNING:
            emoji = '⚠️'
        elif record.levelno == logging.ERROR:
            emoji = '❌'
        elif record.levelno == logging.DEBUG:
            emoji = '🤟'

        format_message = f'{emoji} {record.message}'

        # rewrite message
        record.message = format_message

        # remove useless path
        record.pathname = record.pathname.replace(settings.BASE_DIR + '/', '')
        if record.pathname.startswith('venv'):
            record.pathname = ''
            record.funcName = ''
            record.lineno = 0

        result = super().formatMessage(record)
        return result

# 日志
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    "formatters": {
        'default': {
            # 自定义格式
            '()': 'corelib.loggings.formatters.EmojiLogFormatter',
            'format': '[%(asctime)s][%(pathname)s(%(funcName)s):%(lineno)d]%(message)s'
        },
    },
    'handlers': {
        # 默认的
        'default': {
            'level': 'INFO',
            'class': 'logging.handlers.RotatingFileHandler',  # 按文件大小切割日志
            'filename': LOG_FILE,  # 日志文件
            'formatter': 'default',
            'maxBytes': 1024 * 1024 * 50,  # 每个文件大小
            'backupCount': 0,
        },
        'rq': {
            'level': 'INFO',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': RQ_LOG_FILE,
            'formatter': 'default',
            'maxBytes': 1024 * 1024 * 50,
            'backupCount': 0,
        },
    },
    'loggers': {
        # root logger
        '': {
            'handlers': ['default', ],
            'level': 'INFO',
            'propagate': True,
        },
        'django': {
            'handlers': ['default'],
            'level': 'INFO',
            'propagate': True,
        },
        'django.request': {
            'handlers': ['default', ],
            'level': 'INFO',
            'propagate': False,
        },
        'django.server': {
            'handlers': ['default', ],
            'level': 'INFO',
            'propagate': False,
        },
        'django.db.backends': {
            'handlers': ['default'],
            'level': 'DEBUG',
            'propagate': False,
        },
        'rq_scheduler.scheduler': {
            'handlers': ['rq', ],
            'level': 'INFO',
            'propagate': False,
        },
        "rq.worker": {
            "handlers": ["rq", ],
            "level": "INFO",
            'propagate': False,
        },
    },
}

propagate参数显得尤为重要

Sentry

日志错误处理，早期的Raven使用方式不建议

# settings.py
# 这里不使用SENTRY_DSN
SENTRY_DSN_URL = 'DSN_URL'
sentry_sdk.init(
    SENTRY_DSN_URL,
    integrations=[
        LoggingIntegration(level=logging.WARNING,
                           event_level=logging.WARNING),
        DjangoIntegration(),
        RedisIntegration(),
        RqIntegration(),
    ]
)

Django RQ

相当于将代码序列化到redis中，然后由任意的一台服务器获取执行，序列化中不能包含非基本对象。

Redis Queue

# settings.py
RQ = {
    # 自定义Worker
    'WORKER_CLASS': 'corelib.rq.SentryWorker'
}

自定义RQ Workder，目的是为了添加自定义的Sentry（否则走的是RQ自定义的Sentry）

import rq
import logging
import sentry_sdk
from sentry_sdk.integrations.django import DjangoIntegration
from sentry_sdk.integrations.rq import RqIntegration
from sentry_sdk.integrations.redis import RedisIntegration
from sentry_sdk.integrations.logging import LoggingIntegration
from django.conf import settings


class SentryWorker(rq.Worker):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        sentry_sdk.init(
            # 如果使用SENTRY_DSN，会在最后的时候执行RQ自己的初始化，导致不能自定义，所以在settings中使用SENTRY_DSN_URL，然后在这里自己重新初始化
            settings.SENTRY_DSN_URL,
            integrations=[
                LoggingIntegration(level=logging.WARNING,
                                   event_level=logging.WARNING),
                DjangoIntegration(),
                RedisIntegration(),
                RqIntegration(),
            ]
        )
        self.log_result_lifespan = False

Gunicorn

WSGI

提高Django的并发，创建多个worker（即多个unix进程）

# Run gunicorn with config
venv/bin/gunicorn proc_name.wsgi:application -c gunicorn.conf.py

Gevent：协程框架

协程：使用协程进行io异步提高并发

Config详情

# gunicorn.conf.py
import multiprocessing

proc_name = 'proc_name'
bind = "0.0.0.0:8888"
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = 'gevent'
worker_connections = 2000 # 最大连接数


pidfile = f'pids/{proc_name}.pid'
accesslog = 'logs/gunicorn.access.log'
errorlog = 'logs/gunicorn.error.log'

Supervisor

进程管理，管理Gunicorn和RQ进程，检测异常退出，会自动重启。

Config

; Sample supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html
;
; Notes:
;  - Shell expansion ("~" or "$HOME") is not supported.  Environment
;    variables can be expanded using this syntax: "%(ENV_HOME)s".
;  - Quotes around values are not supported, except in the case of
;    the environment= options as shown below.
;  - Comments must have a leading space: "a=b ;comment" not "a=b;comment".
;  - Command will be truncated if it looks like a config file comment, e.g.
;    "command=bash -c 'foo ; bar'" will truncate to "command=bash -c 'foo ".
;
; Warning:
;  Paths throughout this example file use /tmp because it is available on most
;  systems.  You will likely need to change these to locations more appropriate
;  for your system.  Some systems periodically delete older files in /tmp.
;  Notably, if the socket file defined in the [unix_http_server] section below
;  is deleted, supervisorctl will be unable to connect to supervisord.

[unix_http_server]
file=socks/supervisor.sock   ; the path to the socket file

[supervisord]
logfile=logs/supervisord.log ; main log file; default $CWD/supervisord.log
pidfile=pids/supervisord.pid ; supervisord pidfile; default supervisord.pid

; The rpcinterface:supervisor section must remain in the config file for
; RPC (supervisorctl/web interface) to work.  Additional interfaces may be
; added by defining them in separate [rpcinterface:x] sections.

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

; The supervisorctl section configures how supervisorctl will connect to
; supervisord.  configure it match the settings in either the unix_http_server
; or inet_http_server section.

[supervisorctl]
serverurl=unix://socks/supervisor.sock ; use a unix:// URL  for a unix socket

; The sample program section below shows all possible program subsection values.
; Create one or more 'real' program: sections to be able to control them under
; supervisor.

[program:proc_name]
command=venv/bin/gunicorn music_album_server.wsgi:application -c gunicorn.conf.py            ; the program (relative uses PATH, can take args)
process_name=%(program_name)s ; process_name expr (default %(program_name)s)
numprocs=1                    ; number of processes copies to start (def 1)
autostart=true                ; start at supervisord start (default: true)
startsecs=1                   ; # of secs prog must stay up to be running (def. 1)
startretries=3                ; max # of serial start failures when starting (default 3)
autorestart=true        ; when to restart if exited after running (def: unexpected)
stopasgroup=true             ; send stop signal to the UNIX process group (default false)
killasgroup=true             ; SIGKILL the UNIX process group (def false)
redirect_stderr=true          ; redirect proc stderr to stdout (default false)
stdout_logfile=logs/supervisord_server.log        ; stdout log path, NONE for none; default AUTO
stderr_logfile=AUTO        ; stderr log path, NONE for none; default AUTO
environment=DJANGO_SETTINGS_MODULE="proc_name.settings_test"       ; process environment additions (def no adds)


[program:rqworker]
command=venv/bin/python manage.py rqworker default
process_name=%(program_name)s_%(process_num)02d ; process_name expr (default %(program_name)s)
numprocs=10                    ; number of processes copies to start (def 1)
startsecs=1                   ; # of secs prog must stay up to be running (def. 1)
startretries=3                ; max # of serial start failures when starting (default 3)
stopsignal=TERM               ; signal used to kill process (default TERM)
autostart=true                ; start at supervisord start (default: true)
autorestart=true        ; when to restart if exited after running (def: unexpected)
stopasgroup=true             ; send stop signal to the UNIX process group (default false)
killasgroup=true             ; SIGKILL the UNIX process group (def false)
redirect_stderr=true          ; redirect proc stderr to stdout (default false)
stdout_logfile=logs/supervisord_rqworker.log        ; stdout log path, NONE for none; default AUTO
stderr_logfile=AUTO        ; stderr log path, NONE for none; default AUTO
environment=DJANGO_SETTINGS_MODULE="proc_name.settings_test"       ; process environment additions (def no adds)

# 启动
supervisord -c supervisord_test.conf

supervisorctl -c supervisord_test.conf status
supervisorctl -c supervisord_test.conf status proc_name

# Gunicorn推荐重启方式
supervisorctl -c supervisord_test.conf signal HUP all

supervisorctl -c supervisord_test.conf reload

supervisorctl -c supervisord_test.conf restart

supervisorctl -c supervisord_test.conf start

Nginx

由于历史原因，项目由PHP作为V1版本开发，V2版本全部迁移到Django，所以采用Nginx作为端口转发

Fabric

脚本作为CI/CD工具，更新服务器代码并重启服务器，好处是可以手动重启服务器，缺点是不自动化。

AB测试

被测试服务器信息

单台服务器内网测试

# gunicorn.conf.py
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = 'gevent'
worker_connections = 2000

直接返回JSON，无任何操作

参数设置：-t 10 -c X

ab -H 'Accept-Encoding: gzip' -t 10 -c X -r http://
QPS基本在1000以上，随着并发用户的量变多，平均用户等待时间变长。

参数设置：-n X -c X

ab -H 'Accept-Encoding: gzip' -n X -c X -r http://
QPS基本在800以上，随着并发用户的量变多，平均用户等待时间变长。

数据库查询单条数据、Redis get数据，返回JSON

参数设置：-t 10 -c X

ab -H 'Accept-Encoding: gzip' -t 10 -c X -r http://
QPS基本在250以上，随着并发用户的量变多，平均用户等待时间变长。

这里虽然显示很多错误，但是Gunicorn日志中返回的都是200，原因不详

参数设置：-n X -c X

ab -H 'Accept-Encoding: gzip' -n X -c X -r http://
QPS基本在250以上，随着并发用户的量变多，平均用户等待时间变长。

这里虽然显示很多错误，但是Gunicorn日志中返回的都是200，原因不详

LVS负载（轮询），Nginx，4台服务器

# gunicorn.conf.py
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = 'gevent'
worker_connections = 2000

数据库查询单条数据、Redis get数据，返回JSON

参数设置：-t 10 -c 1000

思考

项目中并未使用微服务架构，也并没有使用docker进行部署，也未使用k8s，可以说是最小的实践方式。
用最简单的方式测试想法是否可以。