个人笔记

专注互联网

SSH隧道

ssh常用来做远程ssl登录,通过添加公钥到~/.ssh/authorized_keys还可以实现无密登录

其他小tip

直接执行命令

king@DESKTOP-89R692H:/home/king$ ssh king@192.168.1.2 pwd
king@192.168.1.2's password:
/home/king

拷贝文件

king@DESKTOP-89R692H:/home/king$ scp test.sh king@192.168.1.2:/tmp
king@192.168.1.2's password:
test.sh 100% 124 0.1KB/s 00:00
king@DESKTOP-89R692H:/home/king$ scp king@192.168.1.2:/tmp/test.sh ./test.sh
king@192.168.1.2's password:
test.sh 100% 124 0.1KB/s 00:00

密码参数

在不适用公钥无密登录的时候,密码必需手动输入,无法通过命令行直接带进去

对此,可以使用开源软件putty来支持,后者有发布Windows版本,不过可以直接在Linux下编译

具体用法参考http://qjw.qiujinwu.com/blog/2013/01/28/putty

本地端口转发

假定host1是本地主机,host2是远程主机。由于种种原因,这两台主机之间无法连通。但是,另外还有一台host3,可以同时连通前面两台主机。因此,很自然的想法就是,通过host3,将host1连上host2。

本地端口:目标主机:目标主机端口 -p 跳板机端口 跳板机username@跳板机host

端口转发的一个类似场景如nginx

只留一个对外的主机,并且开放80端口,通过nginx根据ServerName/Path来分发到upsteam(本机/其他主机的端口),

docker

默认情况docker的端口对于host是不可访问的,当然-p可以将端口share出来

这里模拟一种场景,测试服务器只开放了22/80端口,想通过ssh隧道访问其他端口

准备文件

king@king:~/tmp/docker$ ls
authorized_keys dockerfile sources.list

authorized_keys存host的公钥,文件来自~/.ssh/id_rsa.pub,sources.list设置(比如阿里云)的源,这里我们用它来实现无密登录docker的ssh

dockerfile

FROM ubuntu:16.04

MAINTAINER qiujinwu@gmail.com

RUN mkdir -p /app
WORKDIR /app

COPY sources.list /etc/apt

RUN apt-get update \
&& apt-get install openssh-server netcat iputils-ping telnet net-tools -y \
&& apt-get clean -y \
&& apt-get autoclean

RUN mkdir /var/run/sshd
RUN echo 'root:1' |chpasswd
RUN mkdir /root/.ssh/
COPY authorized_keys /root/.ssh/

EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

build docker镜像

docker build -t sshd:0.1 ~/tmp/docker/

运行

# 进入交互shell,再手动启动sshd
docker run -it --rm -p 10022:22 --name sshd sshd:0.1 /bin/bash
# 自动启动sshd
docker run -it --rm -p 10022:22 --name sshd sshd:0.1
# 关闭(另起shell)
docker stop sshd

echo 服务器

rm -f /tmp/f; mkfifo /tmp/f
cat /tmp/f | nc -l -p 40001 127.0.0.1 > /tmp/f

到此,我们可以启动一个进入shell的docker镜像,然后/etc/init.d/sshd start启动sshd,最后在shell中启动echo服务器

king@king:~$ docker run -it --rm -p 10022:22 --name sshd sshd:0.1 /bin/bash
root@2f661b626247:/app# /etc/init.d/ssh start
* Starting OpenBSD Secure Shell server sshd [ OK ]
root@2f661b626247:/app# rm -f /tmp/f; mkfifo /tmp/f
root@2f661b626247:/app# cat /tmp/f | nc -l -p 40001 127.0.0.1 > /tmp/f
root@2f661b626247:/app#

端口转发

# -N参数,表示只连接远程主机,不打开远程shell
# -T参数,表示不为这个连接分配TTY
ssh -NT -L localhost:40001:localhost:40001 -p 10022 root@127.0.0.1

这是,本机的sshd会监听40001端口,并且转交到本机的10022端口(docker的22端口),然后docker会交给自己的40001端口

接下来在本机运行客户端

king@king:~$ nc 127.0.0.1 40001
aa
aa
bbb
bbb

跨主机

前面的例子第三方的跳板机收到请求后,转发到自己(localhost)的其他端口,也可以转发到其他主机

运行目标机docker

king@king:~/.ssh$ docker run -it --rm --name sshd2 sshd:0.1 /bin/bash
root@bea2bbfe3625:/app# rm -f /tmp/f; mkfifo /tmp/f
# 这里不指定0.0.0.0
root@bea2bbfe3625:/app# cat /tmp/f | nc -l -p 40001 > /tmp/f

运行跳板机docker, 暴露端口22到10022,并且链接到目标机(使用别名up)

docker run -it --rm -p 10022:22 --link=sshd2:up --name=sshd sshd:0.1

参考http://qjw.qiujinwu.com/blog/2013/12/22/nc_server

运行宿主机 ,将本机40001的数据转发到跳板机docker,然后再转发到名为up的docker的40001端口

ssh -NT -L localhost:40001:up:40001 -p 10022 root@127.0.0.1

远程端口转发

host1与host2之间无法连通,必须借助host3转发。但是,特殊情况出现了,host3是一台内网机器,它可以连接外网的host1,但是反过来就不行,外网的host1连不上内网的host3。这时,”本地端口转发”就不能用了,怎么办?
解决办法是,既然host3可以连host1,那么就从host3上建立与host1的SSH连接,然后在host1上使用这条连接就可以了。

外网端口:目标主机:目标主机端口 -p 外网ssh端口 外网机username@外网host

继续使用docker模拟

默认的bridge模式, host机会有一个x.x.x.1的ip和docker机器联通

运行目标机器

king@king:~/.ssh$ docker run -it --rm --name sshd2 sshd:0.1 /bin/bash
root@556f737ec232:/app# rm -f /tmp/f; mkfifo /tmp/f
root@556f737ec232:/app# cat /tmp/f | nc -l -p 40001 > /tmp/f

运行跳板机,通知host机监听40001端口,并且转发给自己,然后自己再转发给up机器的40001端口

king@king:~/tmp/docker$ docker run -it --rm --link=sshd2:up --name=sshd sshd:0.1 /bin/bash
root@fd1b60b1b84e:/app# ssh -NT -R 40001:up:40001 -p 22 king@172.17.0.1
king@172.17.0.1's password:

在host机执行测试程序即可

king@king:~$ nc 127.0.0.1 40001
aa
aa
bbb
bbb

SSL隧道

在本地绑定一个端口,发送往这个端口的数据都通过ssh到远程机器出去,本质上就是socks代理

ssh -NT -D 1080 king@192.168.1.219

浏览器可以安装个switchyomega工具,然后设置socks5代理指向 localhost:1080,就可以让浏览器所有数据都从192.168.1.219出去,从而做些羞羞的事情

参考

  1. http://www.ruanyifeng.com/blog/2011/12/ssh_port_forwarding.html
  2. http://qjw.qiujinwu.com/blog/2013/12/22/nc_server
  3. http://www.cnblogs.com/wellbye/p/3163966.html

telegraf/influxdb/grafana初探

安装

influxdb/grafana直接docker运行最简单

// 下载镜像
docker pull influxdb
docker pull grafana/grafana
# 运行(第一次)
docker run -p 8086:8086 --name influxdb -v $PWD:/var/lib/influxdb influxdb
docker run -p 3000:3000 --name grafana grafana/grafana
# 创建了容器之后,由于指定了自定义name,后续可以根据这个name直接启动
docker start influxdb
docker start grafana

telegraf直接安装

下载地址 https://github.com/influxdata/telegraf/releases

sudo dpkg -i /home/king/telegraf_1.4.5-1_amd64.deb
# 安装客户端
sudo apt-get install influxdb-client
# docker也是可以的
# https://hub.docker.com/_/telegraf/

influxdb

基本概念

  1. dababase 数据库
  2. measurement,数据库表
  3. points,类比数据表行
  4. time, 时间戳,由于是时序数据库,每个数据表都会有一列时间戳的列
  5. field, 字段数据,没有索引,通常根据时间来呈现
  6. tag, 字段的标签,有索引,可以以此进行过滤
  7. series, a series is the collection of data that share a retention policy, measurement, and tag set,

field和tag

两者除了索引的区别外,本质上是定位的差异,举个例子,有数据

时间戳 姓名 城市 收入 支出

收入支出定位于field,没有实际的规律,一般不会用它来做顾虑(传统的关系数据库,常见的可以根据范围过滤搜索),而姓名城市定位于Tag,可以理解为这一行(point)的属性,我们会有诸如

  1. 所有人的收入曲线图
  2. 广州、深圳等各个城市的收入,支出曲线图
  3. 甚至只看广州的张三的支出曲线图

Tag一般在一个有限的集合中选取,而不是field那样无明显的规律

写入数据

# 创建数据库
curl -i -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE telegraf"
# 写入数据,最后的时间戳可以省略,自动用当前时间(test: measurement名称)
curl -i -XPOST 'http://localhost:8086/write?db=telegraf' --data-binary \
'test,tag1=server01,tag2=us-west field1=0.64,field2=11 1434055562000000000'

也可以使用cli工具

king@king:~/tmp$ influx 
Visit https://enterprise.influxdata.com to register for updates,
InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 1.4.2
InfluxDB shell 0.10.0
> show databases;
name: databases
---------------
name
_internal
telegraf

> use telegraf;
Using database telegraf
> show measurements;
name: measurements
------------------
name
money
processes
swap
system
test

> show series from money
key
money,city=上海,name=张三
money,city=北京,name=刘七

> select * from money limit 2
name: money
-----------
time city in name out
1512445152435671225 深圳 28 赵六 1
1512445153519585060 上海 76 李四 95

简单的数据源

不依赖于telegraf采集器,在程序中,也有非常多的api可选,参见https://docs.influxdata.com/influxdb/v0.9/clients/api/

#!/bin/bash

url="http://localhost:8086/write?db=telegraf"
names=('张三' '李四' '王五' '赵六' '刘七')
citys=('深圳' '广州' '北京' '上海' '成都')

doWrite(){
curl -i -XPOST "${url}" --data-binary "money,name="${1}",city="${2}" in="${3}",out="${4}""
}

for((i=0;i<10000;i++))
do
in=$(($(echo $RANDOM) % 100))
out=$(($(echo $RANDOM) % 100))

cityid=$(($(echo $RANDOM) % 5))
nameid=$(($(echo $RANDOM) % 5))
doWrite ${names[nameid]} ${citys[cityid]} "${in}" "${out}"
sleep 1
done

Telegraf

telegraf是一个数据采集器,自身有一些基本的采集功能,比如cpu、内存等。支持插件的设计使得采集范围非常广泛,支持列表见https://github.com/influxdata/telegraf#input-plugins

具体的配置见官方文档,为了对接influxdb,我们先需要修改/etc/telegraf/telegraf.conf

# Configuration for influxdb server to send metrics to
[[outputs.influxdb]]
## The full HTTP or UDP URL for your InfluxDB instance.
##
## Multiple urls can be specified as part of the same cluster,
## this means that only ONE of the urls will be written to each interval.
# urls = ["udp://localhost:8089"] # UDP endpoint example
urls = ["http://localhost:8086"] # required
## The target database for metrics (telegraf will create it if not exists).
database = "telegraf" # required

直接运行telegraf命令启动服务器(为什么没有/etc/init.d/xx)

grafana

grafana本身支持很多后端,包括influxdb,它是启动之后再进行参数配置。

首先配置数据源,influxdb参考http://docs.grafana.org/features/datasources/influxdb/

然后添加dashboard-panel。在panel的metrics里有各种参数(本质上就是一个sql的语义),以及一些聚合函数支持

完善权限控制

这三个东西和elk技(全)术(家)栈(桶)有些有意思的对比,这里有一个grafana和kibana的比较

参考

  1. http://www.jianshu.com/p/dfd329d30891

antd随笔

antd https://ant.design/index-cn

环境

本着学习的目的,最开始自己从零折腾,不过后来放弃了,学习从https://github.com/zuiidea/antd-admin开始

可以直接删除业务/页面代码,保留配置继续开发,后者体验网址http://antd-admin.zuiidea.com

html5/es6+/react

大量使用es6语法

简化的import/export,以及大括号语法,自动导出子模块(变量)

import {request, config} from 'utils'
const {api} = config;

异步函数/async

export async function loginCb(data) {
return request({
url: api.userLoginCB,
method: 'get',
data,
})
}

生成器函数/yield/*函数

yield put({
type: 'querySuccess',
payload: {
data: data,
},
})

* query({
url,
payload,
}, {call, put}) {
const {data} = yield call(url, payload);
yield put({
type: 'querySuccess',
payload: {
data: data,
},
})
},

箭头函数

export default connect(({ detail, loading }) => ({ state:detail, loading }))(Detail)

const handleOk = () => {
validateFields((errors) => {
if (errors) {
return
}
const data = {
...item,
...getFieldsValue(),
};
onOk(data)
})
};

// 多重
const handleChange = (count) => (fileList) => {
// do sth
};

合并对象/…

const data = {
...item,
...getFieldsValue(),
};

const/let声明变量/常亮

(不)等于 ===/!==

map/filter

window.localStorage保存本地配置

React

react组件有三种声明方式

antd大量使用函数式定义的写法,这种写法是废代码不多,但是不支持状态和react事件

import React from 'react'
import { config } from 'utils'
import styles from './Footer.less'

const Footer = () => (<div className={styles.footer}>
{config.footerText}
</div>)

export default Footer

es5原生方式React.createClass定义的组件

es6形式的extends React.Component定义的组件

跳转

跳转可以用history接口https://developer.mozilla.org/en-US/docs/Web/API/History_API

或者更高层的history库 https://github.com/ReactTraining/history

不过推荐dispatch一个routerRedux异步任务的方式

dispatch(routerRedux.push({
pathname: buildUrl(route.apartmentBatchPrice,{id:record.id}),
}))

React router的迁移问题

React router 2到4接口改变了很多,antd/dva版本上也有出现很多问题,参见https://github.com/gmfe/Think/issues/6

新版本不再有query成员,不过可以手动处理下

const queryString = require('query-string');
location.query = queryString.parse(location.search);

参数

跳转时,一般使用url的query参数传递参数,不过若使用ReactTraining/history 库,可以借助location.state

  1. location.pathname - The path of the URL
  2. location.search - The URL query string
  3. location.hash - The URL hash fragment
  4. location.state - Some extra state for this location that does not reside in the URL (supported in createBrowserHistory and createMemoryHistory)
  5. location.key - A unique string representing this location (supported in createBrowserHistory and createMemoryHistory)

routerRedux.push的参数就是一个location对象

Model复用

借助dva提供的modelExtend,可以复用model的变量和方法,

import modelExtend from 'dva-model-extend'

const model = {
reducers: {
updateState (state, { payload }) {
return {
...state,
...payload,
}
},
},
};

const pageModel = modelExtend(model, {
state: {
list: [],
pagination: {
showSizeChanger: false,
showQuickJumper: false,
showTotal: total => `共 ${total} 条记录`,
current: 1,
total: 0,
},
},

reducers: {
querySuccess (state, { payload }) {
const { list, pagination } = payload
return {
...state,
...{list:list},
pagination: {
...state.pagination,
...pagination,
},
}
},
},

});


module.exports = {
model,
pageModel,
};

和C++的继承有几分相似的是,modelExtend的继承可以继续继承,也可以多继承

const pageModel = modelExtend(model, model2 {
// ...
}

service优化

service定义了调用后端Api的接口,每个route的service结构都大同小异,考虑到常用的就CRUD,对于同一个后端,使用的http method/参数定义也相同,所以可以抽象以下。

下面的代码架设了create/query/detail/update/enable/disable/remove等常用方法的请求格式

import {request, config} from 'utils'
import {buildUrl} from '../utils/url'

export function createService(keys) {
let res = {};
let exclude = {}
if (keys["create"]) {
res["create"] =
async function create(data, param=null) {
return request({
url: buildUrl(keys.create, param),
method: 'post',
data,
})
}
exclude["create"] = null
}

if (keys["query"]) {
res["query"] =
async function query(data, param=null) {
return request({
url: buildUrl(keys.query, param),
method: 'get',
data,
})
}
exclude["query"] = null
}

if (keys["detail"]) {
res["detail"] =
async function detail(param = null) {
return request({
url: buildUrl(keys.detail, param),
method: 'get',
})
}
exclude["detail"] = null
}

if (keys["update"]) {
res["update"] =
async function update(data, param=null) {
return request({
url: buildUrl(keys.update, param),
method: 'put',
data: data,
})
}
exclude["update"] = null
}

if (keys["enable"]) {
res["enable"] =
async function enable(param = null) {
return request({
url: buildUrl(keys.enable, param),
method: 'post',
})
}
exclude["enable"] = null
}

if (keys["disable"]) {
res["disable"] =
async function disable(param = null) {
return request({
url: buildUrl(keys.disable, param),
method: 'post',
})
}
exclude["disable"] = null
}

if (keys["remove"]) {
res["remove"] =
async function remove(param = null) {
return request({
url: buildUrl(keys.remove, param),
method: 'delete',
})
}
exclude["remove"] = null
}

const methods = {
"post": null,
"put": null,
"get": null,
"delete": null,
"patch": null,
}

for (const key in keys) {
if (keys.hasOwnProperty(key)){
// 排除上面的通用方法
if(key in exclude) continue
// 必须是对象
if(typeof keys[key] === 'object'){
const rt = keys[key]
// 合法的方法
if(!rt.method in methods) {
continue
}

res[key] =
async function create(data, param=null) {
return request({
url: buildUrl(rt.url, param),
method: rt.method,
data,
})
}
}
}
}

return res
}

import {request, config} from 'utils'
import {createService} from "./common"

exports.services = createService(config.api.user);
user: {
"query": `${APIV1}/users/`,
"update": `${APIV1}/users/:id`,
"detail": `${APIV1}/users/:id`,
"enable": `${APIV1}/users/:id/enable`,
"disable": `${APIV1}/users/:id/disable`
},

一些特殊的方法,则简单描述即可

import {request, config} from 'utils'
import {createService} from "./common"

let routes = config.api.price
if(typeof routes["generate"] != 'object'){
routes['generate'] = {
"method": "post",
"url": routes['generate']
}
}

exports.services = createService(routes);
price: {
"create": `${APIV1}/:id/prices`,
"update": `${APIV1}/:id/prices`,
"query": `${APIV1}/:id/prices`,
"generate": `${APIV1}/:id/prices/generate`,
}

Url

大量使用path-to-regexp

有用到几种场景

判断是否是我们需要的url

setup({dispatch, history}) {
history.listen((location) => {

const match = pathToRegexp(route.apartments).exec(location.pathname);
if (match) {
location.query = queryString.parse(location.search);
const payload = location.query || {page: 1, count: countPePage};
dispatch({
type: 'query',
payload,
})
}
})
},

构建url用于跳转,解析url中的参数

import pathToRegexp from 'path-to-regexp'

module.exports = {
buildUrl:(url, param) => {
if(!param) return url;

const toPath = pathToRegexp.compile(url);
return toPath(param);
},

parseUrl:(url,route) => {
const match = pathToRegexp(route).exec(url)
if(!match) return {}

let urlDatas = {};
let index = 1;
const keys = pathToRegexp.parse(route)
keys.forEach(function(item){
if(typeof item !== 'object'){
return
}
urlDatas[item.name] = match[index]
index ++
});
return urlDatas
}
};

yield put(routerRedux.push({
pathname: buildUrl(route.storeDetailIndex,{id:store.id,tab:route.storeDetailTab.rule}),
}))
const urlDatas = parseUrl(location.pathname, current.route);
pathArray.forEach(function (item) {
if(item === current || !item.route) return;
item.route2 = buildUrl(item.route,urlDatas)
})

dva-loading

通过绑定一个redux-saga异步请求来实现UI的加载中的效果

关于dva以及react全家桶的一些概念,参见https://github.com/dvajs/dva/blob/master/README_zh-CN.md

import createLoading from 'dva-loading'
// 1. Initialize
const app = dva({
...createLoading({
effects: true,
}),
});
export default connect(({ detail, loading }) => ({ state:detail, loading }))(Detail)
<Button type="primary" size="large" loading={loading.effects.login}>
登录
</Button>

antd很多组件都集成了loading属性。例如buttonTable

当页面不存在这些支持loading的控件,或者不合适使用时,antd-admin提供了一个全局的Loader控件

const LoginCB = ({location, dispatch, loading}) => {
return (
<div>
<Loader fullScreen spinning={loading.effects['login_cb/login']} />
</div>
)
}

表单

<Form.Item {...props}>
{children}
</Form.Item>

form基于rc-form,一些选项可以参考https://github.com/react-component/form#option-object

经过 getFieldDecorator 包装的控件,表单控件会自动添加 value(或 valuePropName 指定的其他属性) onChange(或 trigger 指定的其他属性),数据同步将被 Form 接管,这会导致以下结果:

  1. 你不再需要也不应该用 onChange 来做同步,但还是可以继续监听 onChange 等事件。
  2. 你不能用控件的 value defaultValue 等属性来设置表单域的值,默认值可以用 getFieldDecorator 里的 initialValue
  3. 你不应该用 setState,可以使用 this.props.form.setFieldsValue 来动态改变表单值。

自定义控件适配

参考https://ant.design/components/form/#components-form-demo-customized-form-controls

关键两点

  1. 初始值从props.value获取,这个值从getFieldDecoratorinitialValue获取
  2. 当控件value值改变时,通过props.onChange获取到触发getFieldDecorator的回调,并触发之
import React from 'react'
import {Select} from 'antd';
import {connect} from 'dva'
import {Icon} from 'antd';
import PropTypes from 'prop-types';
const {Option} = Select;

class IDSelect extends React.Component {
constructor(props) {
super(props);

const value = this.props.value || [];
this.state = {
current_value: undefined,
list: [],
};
}

componentWillReceiveProps(nextProps) {
// Should be a controlled component.
if ('value' in nextProps) {
// ----- mark------
const value = nextProps.value;
if(value === undefined) return

if (!value){
this.setState({current_value: null});
}else{
this.setState({current_value:String(value)});
}
}

if ('list' in nextProps) {
const list = nextProps.list;
if(list === undefined) return
this.setState({list});
}
}

render() {
const handleChange = (value) => {
this.setState({current_value:value});
// ----- mark------
const onChange = this.props.onChange;
if (onChange) {
onChange(parseInt(value));
}
}

const {list,value,...props} = this.props
return (
<span>
{this.state.list.length > 0 && <Select
{...props}
defaultValue={this.state.current_value}
onChange={handleChange}
>
{this.state.list.map(function (object, i) {
return <Option value={String(object.id)} key={i}>{object.name}</Option>;
})}
</Select>}
</span>
);
}
}

IDSelect.propTypes = {
value: PropTypes.number,
list: PropTypes.arrayOf(PropTypes.object),
};

module.exports = {
IDSelect,
};
<FormItem>
{getFieldDecorator('user_id', {
initialValue: state.editType === 'create' ? null : data.user_id,
rules: [
{
required: true,
}
],
})(
<IDSelect
disabled={state.editType !== 'create'}
list={state.users}
/>
)}
</FormItem>

提交

提交时,使用form.validateFields做校验,参数中values就是包含所有表单参数的map,map的key是getFieldDecorator的第一个参数。例如上面的user_id

某些控件返回的数据并非后端需要的格式(例如DatePicker返回一个moment,Select返回一个String类型的ID),在提交到后端的API之前,在这里做格式的转换

const handleSubmit = (e) => {
e.preventDefault();
form.validateFields((err, values) => {
if (!err) {
// do submit
}
});
};

校验

规则见https://github.com/yiminghe/async-validator

const checkCoordinate = (rule, value, callback) => {
if(!parseCoordinate(value)){
callback("错误的经纬度");
return
}

callback();
};

<FormItem>
{getFieldDecorator('coordinate', {
initialValue: state.editType === 'create' ? "" : String(data.lng) + ',' + String(data.lat),
rules: [
{
required: true,
min: 1,
max: 64,
},
{validator: checkCoordinate},
],
})(<Input placeholder="请输入经纬度" style={{width: '100%'}}/>)}
</FormItem>
module.exports = {
parseCoordinate:(value) => {
const values = value.split(",");
if (values.length !== 2) {
return null;
}

try {
return {
lng: parseFloat(values[0]),
lat: parseFloat(values[1])
};
}
catch (e) {
return null;
}
}
}

layout

form有个全局的layout,参见form的layout参数。'horizontal'|'vertical'|'inline'

若需要嵌套(例如每个input一行,某些行有多个input),则需要借助Col来控制

<FormItem
label="位置"
required
{...formItemLayout}
>
<Col span={6}>
<FormItem>
{getFieldDecorator('param1', {
initialValue: state.editType === 'create' ? "" : data.param1,
rules: [
{
required: true,
min: 1,
max: 32
},
],
})(<Input style={{width: '100%'}}/>)}
</FormItem>
</Col>
<Col span={6}>
<FormItem>
{getFieldDecorator('param2', {
initialValue: state.editType === 'create' ? "" : data.param2,
rules: [
{
required: true,
type: "integer"
},
],
})(<InputNumber style={{width: '100%'}}/>)}
</FormItem>
</Col>
</FormItem>

关于required的红点

若某个表单是必选的,左边的label会有个红色的*。但是对于一行有多个表单(只有一个左侧的label)默认情况下,不会有

解决办法是(见上面代码)在最外层的FormItem加上required,内层的无需处理

// 一行多表单
<FormItem
label="位置"
required
{...formItemLayout}
>

富文本编辑器

官方推荐的react-lz-editor相当不错,并且支持markdown和普通的wysiwyg模式,不过有一个需求(粘贴富文本格式)似乎满足不了,

react-draft-wysiwyg也是个不错的选择

import {Editor} from 'react-draft-wysiwyg';
import 'react-draft-wysiwyg/dist/react-draft-wysiwyg.css';
import { EditorState, convertToRaw, ContentState, convertFromHTML } from 'draft-js';

const checkContent = (rule, value, callback) => {
if (value && state.editorState) {
const content = draftToHtml(convertToRaw(state.editorState.getCurrentContent()));
if(content.length < 9 || content.length >= 1024000){
callback("content must be between 2 and 1024000 characters\n");
return
}
}
callback();
};

const uploadImageCallBack = (file) => {
return new Promise(
(resolve, reject) => {
const xhr = new XMLHttpRequest();
xhr.open('POST', '/api/v1/upload');
const data = new FormData();
data.append('file', file);
xhr.send(data);
xhr.addEventListener('load', () => {
let response = JSON.parse(xhr.responseText);
response = { data: { link: response.data.url,alt: "fasdf" } }
resolve(response);
});
xhr.addEventListener('error', () => {
const error = JSON.parse(xhr.responseText);
reject(error);
});
}
);
};

const onEditorStateChange = (editorState) => {
dispatch({
type: `${pageKey}/updateState`,
payload: {
editorState
}
})
};

<Editor
editorState={state.editorState}
wrapperClassName="demo-wrapper"
editorClassName="demo-editor"
toolbar={{
image: { uploadCallback: uploadImageCallBack, alt: { present: false, mandatory: false } },
}}
onEditorStateChange={onEditorStateChange}
/>

百度开源的ueditor非常完善,但是体积非常大,一个精简版UMeditor

文件上传

以下代码配合后端上传接口实现

import React from 'react'
import {Upload} from 'antd';
import {connect} from 'dva'
import { config } from '../utils'
import {Icon} from 'antd';

const UploadButton = () => (
<div>
<Icon type="plus" />
<div className="ant-upload-text">点击上传</div>
</div>
);

const handleChange = (count) => (fileList) => {
// 1. Limit the number of uploaded files
// Only to show two recent uploaded files, and old ones will be replaced by the new
fileList = fileList.slice(-count);

// 2. filter successfully uploaded files according to response from server
fileList = fileList.filter((file) => {
if (file.response) {
if (file.response.code !== 0) {
return false
}
}
return true;
});

// 3. read from response and show file link
fileList = fileList.map((file) => {
if (file.response) {
// Component will show file.url as link
file.url = file.response.data.url;
}
return file;
});

return fileList
};

const handleOneChange = (fileList) => {
return handleChange(1)(fileList);
};

const normOneFile = (e) => {
let res = e;
if (Array.isArray(e)) {
res = e;
} else {
res = e.fileList
}

if (res.length < 1) {
return ""
} else {
return res[0].url;
}
};

const normFile = (e) => {
let res = e;
if (Array.isArray(e)) {
res = e;
} else {
res = e.fileList
}

if (res.length < 1) {
return ""
} else {
return res.map(function(object, i) {
return object.url
});
}
};

class ImageUploader extends React.Component {
constructor(props) {
super(props);

const value = this.props.value || [];
this.state = {
images: value
};
}

componentWillReceiveProps(nextProps) {
// Should be a controlled component.
if ('value' in nextProps && !!nextProps.value) {
const value = nextProps.value;

// 排除uploading的
if(!value) return;

this.setState({
images: [
{
uid: -1,
name: 'test.png',
status: 'done',
url: value
}
]
});
}
}

render() {
const handleChangeImp = (info) => {
const res = handleOneChange(info.fileList)
this.setState({images: res});

const onChange = this.props.onChange;
if (onChange) {
onChange(res);
}
};

return (<span>
<Upload name="file" listType="picture-card"
fileList={this.state.images} onChange={handleChangeImp} action="/api/v1/upload">
{
this.state.images.length >= 1
? null
: <UploadButton/>
}
</Upload>
</span>);
}
}

class ImageMultiUploader extends React.Component {
constructor(props) {
super(props);

const value = this.props.value || [];
this.state = {
images: value
};
}

componentWillReceiveProps(nextProps) {
// Should be a controlled component.
if ('value' in nextProps && !!nextProps.value) {
const value = nextProps.value;

// 排除uploading的
const newValue = value.filter(function(item) {
return (!item);
});
if(newValue.length > 0) return

let image_urls = [];
value.forEach(function(item, index) {
image_urls.push({
uid: -(index + 1),
name: 'test' + index + '.png',
status: 'done',
url: item
})
});

this.setState({images: image_urls});
}
}

render() {
const handleChangeImp = (count) => (info) => {
const res = handleChange(count)(info.fileList)

this.setState({images: res});

const onChange = this.props.onChange;
if (onChange) {
onChange(res);
}
};

return (<span>
<Upload name="file" listType="picture-card" fileList={this.state.images}
onChange={handleChangeImp(this.props.maxImage)} action="/api/v1/upload">
{
this.state.images.length >= this.props.maxImage
? null
: <UploadButton/>
}
</Upload>
</span>);
}
}

module.exports = {
ImageUploader,
ImageMultiUploader,
normOneFile,
normFile
};
<FormItem
{...formItemLayout}
label="照片"
>
{getFieldDecorator('image_url', {
initialValue: state.editType === 'create' ? "" : data.image_url,
getValueFromEvent: normOneFile,
rules: [
{
required: true,
},
],
})(
<ImageUploader />
)}
</FormItem>

一些不错的三方库

  1. prop-types,检查组件参数
  2. path-to-regexp,url工具库
  3. react-helmet,设置title以及头部信息
  4. nprogress,加载进度条,和dva-loading配合使用
  5. axios,基于promise的http客户端库,dva-admin实现了一个比较完善的request函数
  6. query-string,解析url查询参数
  7. react-amap,高德地图
  8. sprintf.js,类似于C printf的占位符打印实现
  9. ReactInlineEdit,双击编辑控件
  10. qs,A querystring parser with nesting support
  11. lodash,工具库
  12. 其他https://ant.design/docs/react/recommendation-cn
if (lastHref !== href) {
NProgress.start()
if (!loading.global) {
NProgress.done()
lastHref = href
}
}

日期

antd使用moment作为日期处理库

获取当月/指定月的区间

import Moment from 'moment';

const getMonthDateRange = (param) => {
const startDate = Moment([param.year(), param.month()]);

// Clone the value before .endOf()
const endDate = Moment(startDate).endOf('month');

// make sure to call toDate() for plain JavaScript date type
return { from: startDate, to: endDate };
}

const getCurMonthRange = () => {
return getMonthDateRange(Moment())
}

module.exports = {
getMonthDateRange,
getCurMonthRange,
}

DatePicker

最新的DatePicker输入输出都是一个moment对象,下面的代码做了个处理,输入输出都是如2017-12-04的这种格式

import React from 'react'
import {DatePicker} from 'antd';
import {connect} from 'dva'
import {config} from '../utils'
import {Icon} from 'antd';
import moment from 'moment';
import PropTypes from 'prop-types';
const RangePicker = DatePicker.RangePicker;

class StringDatePicker extends React.Component {
constructor(props) {
super(props);

const value = this.props.value || [];
this.state = {
current_value: undefined
};
}

componentWillReceiveProps(nextProps) {
// Should be a controlled component.
if ('value' in nextProps) {
const value = nextProps.value;
if(value === undefined) return

if (!value){
this.setState({current_value: null});
return;
}
const current_value = moment(value, 'YYYY-MM-DD')
this.setState({current_value});
}
}

render() {
const handleChange = (value, dateString) => {
let current_value = null;
if(!!value){
current_value = value.format('YYYY-MM-DD');
}
this.setState({
current_value
})

const onChange = this.props.onChange;
if (onChange) {
onChange(current_value);
}
}

return (
<span>
{this.state.current_value !== undefined && <DatePicker
defaultValue={this.state.current_value}
onChange={handleChange}
showToday={true}
/>}
</span>
);
}
}

StringDatePicker.propTypes = {
value: PropTypes.string
};


class StringRangePicker extends React.Component {
constructor(props) {
super(props);

const value = this.props.value || [];
this.state = {
current_value: undefined
};
}

componentWillReceiveProps(nextProps) {
// Should be a controlled component.
if ('value' in nextProps) {
const value = nextProps.value;
if(value === undefined) return
if(!!value && value.length > 0){
const newValue = value.filter(function(item) {
return (!!item);
});
if(newValue.length !== value.length)
return;
}


if (!value){
this.setState({current_value: []});
return;
}
const current_value = value.map(item=>moment(item, 'YYYY-MM-DD'))
this.setState({current_value});
}
}

render() {
const handleChange = (value, dateString) => {
let current_value = null;
if(!!value){
current_value=value.map(item=>item.format('YYYY-MM-DD'))
}
this.setState({
current_value
})

const onChange = this.props.onChange;
if (onChange) {
onChange(current_value);
}
}

return (
<span>
{this.state.current_value !== undefined && <RangePicker
defaultValue={this.state.current_value}
onChange={handleChange}
showToday={true}
/>}
</span>
);
}
}

StringRangePicker.propTypes = {
value: PropTypes.arrayOf(PropTypes.string)
};

module.exports = {
StringDatePicker,
StringRangePicker,
};

高德地图

import React from 'react'
import {mapKey} from '../../utils/config'
import { Map, Marker } from 'react-amap';

const AMap = ({lng,lat}) => {
const plugins = [
'MapType',
'Scale',
'OverView',
];

return (
<div style={{width:"100%",minHeight:"400px"}}>
<Map amapkey={mapKey}
plugins={plugins}
zoom={15}
center={{longitude: lng, latitude: lat}}
>
<Marker position={{longitude: lng, latitude: lat}} />
</Map>
</div>
)
};

export default AMap

Layout

一个常见的管理系统后台,通常都包含一些共有的组件,例如左侧菜单入口,上面导航条,底部说明,面包屑菜单

antd admin的实现参考https://github.com/zuiidea/antd-admin/tree/master/src/components/Layout

antd admin通过route的一个简单设计,是的每个子页面都无需考虑这些在它外围的基础控件

const Routers = function ({history, app}) {
const error = dynamic({
app,
component: () => import('./routes/error'),
});
const routes = [
{
path: route.users,
models: () => [import('./models/user/index')],
component: () => import('./routes/user/index'),
},{
path: route.userLogin,
component: () => import('./routes/login/'),
}
];

return (
<ConnectedRouter history={history}>
<App>
<Switch>
<Route exact path="/" render={() => (<Redirect to="/user"/>)}/>
{
routes.map(({path, ...dynamics}, key) => (
<Route key={key}
exact
path={path}
component={dynamic({
app,
...dynamics,
})}
/>
))
}
<Route component={error}/>
</Switch>
</App>
</ConnectedRouter>
)
};

所有的Page都包含在App Page里面

/* global window */
import React from 'react'
import NProgress from 'nprogress'
import PropTypes from 'prop-types'
import pathToRegexp from 'path-to-regexp'
import { connect } from 'dva'
import { Layout, Loader } from 'components'
import { classnames, config } from 'utils'
import { Helmet } from 'react-helmet'
import { withRouter } from 'dva/router'
import '../themes/index.less'
import './app.less'
import Error from './error'

const { prefix, openPages,name } = config

const { Header, Bread, Footer, Sider, styles } = Layout
let lastHref

const App = ({ children, dispatch, app, loading, location }) => {
// 顶部进度条
if (lastHref !== href) {
NProgress.start()
if (!loading.global) {
NProgress.done()
lastHref = href
}
}

if (openPages && openPages.includes(pathname)) {
// 对于特殊的页面,比如login,直接返回Page内容,而不在外面包含公用的组件
return (<div>
<Loader fullScreen spinning={loading.effects['app/query']} />
{children}
</div>)
}
return (
<div>
{/* 全局的loading效果 */}
<Loader fullScreen spinning={loading.effects['app/query']} />
{/* html title、配置等 */}
<Helmet>
<title>{ name }</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<link rel="icon" href={logo} type="image/x-icon" />
{iconFontJS && <script src={iconFontJS} />}
{iconFontCSS && <link rel="stylesheet" href={iconFontCSS} />}
</Helmet>
<div className={classnames(styles.layout, { [styles.fold]: isNavbar ? false : siderFold }, { [styles.withnavbar]: isNavbar })}>
// 侧边菜单
{!isNavbar ? <aside className={classnames(styles.sider, [styles.light])}>
{siderProps.menu.length === 0 ? null : <Sider {...siderProps} />}
</aside> : ''}
<div className={styles.main}>
// 顶部导航
<Header {...headerProps} />
// 面包屑菜单
<Bread {...breadProps} />
// 每个页面实际的内容
<div className={styles.container}>
<div className={styles.content}>
{hasPermission ? children : <Error />}
</div>
</div>
// 底部说明
<Footer />
</div>
</div>
</div>
)
}

export default withRouter(connect(({ app, loading }) => ({ app, loading }))(App))

这种设计的好处是可以利用App的model处理一些通用的东西,比如

全局的错误监听(models/app.js)

setup({dispatch, history}) {
// Add a response interceptor
axios.interceptors.response.use(function (response) {
// Do something with response data
return response;
}, function (error) {

const {response} = error
let msg;
let statusCode;
if (response && response instanceof Object) {
statusCode = response.status
if (statusCode === 401) {
// Do something with response error
dispatch({
type: 'unauthorized',
payload: error,
})
}
}

return Promise.reject(error);
});

添加路径/路由信息,以及处理一些自动的跳转(登录成功后,存在cookie访问login页面等)

history.listen((location) => {
dispatch({
type: 'updateState',
payload: {
locationPathname: location.pathname,
locationQuery: queryString.parse(location.search),
},
});

const match = pathToRegexp(route.userLoginCB).exec(location.pathname)
if (!match) {
dispatch({type: 'query'})
}
});

授权检查拦截

* query({
payload,
}, {call, put, select}) {

const {locationPathname, locationQuery} = yield select(_ => _.app);
// 如果不是登录页,那么如果存在就不再查询
const match = pathToRegexp(route.userLogin).exec(locationPathname);
if (!match) {
const {user} = yield select(_ => _.app);
if ("id" in user) {
return
}
}

const {data} = yield call(query, payload);
if (data) {
const menuData = yield call(menusService.services.query);
let user = data
let menu = menuData.data;
let visit = menu.map(item => item.id);
yield put({
type: 'updateState',
payload: {
user,
menu,
},
});

if (locationPathname === route.userLogin) {
if (locationQuery["url"]) {
window.location.assign(locationQuery["url"]);
} else {
yield put(routerRedux.replace({
pathname: indexPage,
}))
}
}
} else if (config.openPages && config.openPages.indexOf(locationPathname) < 0) {
yield put(routerRedux.push({
pathname: route.userLogin,
search: queryString.stringify({
from: locationPathname,
}),
}))
}
},


* unauthorized(action, {put, select}) {
const {locationPathname} = yield select(_ => _.app)
if (locationPathname !== route.userLogin) {
yield put(routerRedux.push({
pathname: route.userLogin,
search: queryString.stringify({
url: window.location.href
}),
}))
}
},

当然一些全局的操作,比如logout处理,左侧菜单栏收房,面包屑菜单处理等也可以在这里一并处理好

面包屑菜单

左侧导航菜单的内容,从后端拉取,后端可以根据权限返回有限的数据

antd-admin的实现见https://github.com/zuiidea/antd-admin/blob/master/src/components/Layout/Menu.js

输入的菜单数据是个数组,在前端组装一个树结构,由于后端通常也是一个配置来写这些菜单,数组结构抽象树结构很难理解,所以可以考虑将输入也接收树结构,利于后端编辑

为了适配面包屑菜单,后端返回的菜单数据,有一种隐藏item,这种item不显示在左边菜单,而仅仅用于面包屑菜单匹配

ant-admin的Bread没有支持导航带参数的链接。比如

[首页] => [用户列表] => [用户详情] => [用户编辑]

一般的做法,用户详情/编辑的地址会有一个user_id,而不是固定的(例如用户列表这种)

基于面包屑菜单所需要的参数在当前page中都存在的假设,可以如下处理

import React from 'react'
import PropTypes from 'prop-types'
import { Breadcrumb, Icon } from 'antd'
import { Link } from 'react-router-dom'
import pathToRegexp from 'path-to-regexp'
import { queryArray } from 'utils'
import styles from './Bread.less'
import {parseUrl,buildUrl} from '../../utils/url'

const Bread = ({ menu, location }) => {
// 匹配当前路由
let pathArray = []
let current;
let match;
for (let index in menu) {
if (menu[index].route) {
match = pathToRegexp(menu[index].route).exec(location.pathname);
if(match){
current = menu[index];
break
}
}
}

const getPathArray = (item) => {
pathArray.unshift(item);
if (item.pid) {
getPathArray(queryArray(menu, item.pid, 'id'))
}
};

if (!current) {
pathArray.push(menu[0] || {
id: 1,
icon: 'laptop',
name: 'Dashboard',
});
pathArray.push({
id: 404,
name: 'Not Found',
})
} else {
getPathArray(current)
}

// 自动填充
if(current){
const urlDatas = parseUrl(location.pathname, current.route);
pathArray.forEach(function (item) {
if(item === current || !item.route) return;
item.route2 = buildUrl(item.route,urlDatas)
})
}

// 递归查找父级
const breads = pathArray.map((item, key) => {
const content = (
<span>{item.icon
? <Icon type={item.icon} style={{ marginRight: 4 }} />
: ''}{item.name}</span>
)
return (
<Breadcrumb.Item key={key}>
{((pathArray.length - 1) !== key && "route" in item)
? <Link to={item.route2 || '#'}>
{content}
</Link>
: content}
</Breadcrumb.Item>
)
})

return (
<div className={styles.bread}>
<Breadcrumb>
{breads}
</Breadcrumb>
</div>
)
}

Bread.propTypes = {
menu: PropTypes.array,
location: PropTypes.object,
}

export default Bread

省市区

基于动态读取后端数据的省市区控件

数据来源腾讯地图LBS

import React from 'react'
import {Cascader} from 'antd';
import {connect} from 'dva'
import { config } from '../utils'

class Geography extends React.Component {
constructor(props) {
super(props);

this.state = {
options: [],
current_province: null,
current_city: null,
current_area: null,
default_value: null
};
}

updateConfig(){
let options = [];
const {provinces,citys,areas} = this.props
const state = this.state

if(!!provinces){
provinces.forEach(function (item) {
options.push({
value: item.id,
label: item.fullname,
isLeaf: false,
type: 'province',
data: item,
})
})
}

let currentCitys = null;
if(!!state.current_province && !!citys) {
options.find(function(province){
if(province.value == state.current_province){
let tmp_citys = []
citys.forEach(function (item) {
tmp_citys.push({
value: item.id,
label: item.fullname,
isLeaf: false,
type: 'city',
data: item,
})
})
currentCitys = tmp_citys;
province.children = tmp_citys;
return true;
}else{
return false;
}
})
}

if(!!currentCitys && !!state.current_city && !!areas){
currentCitys.find(function(city){
if(city.value == state.current_city){
let tmp_areas = []
areas.forEach(function (item) {
tmp_areas.push({
value: item.id,
label: item.fullname,
isLeaf: true,
type: 'area',
data: item,
})
})
city.children = tmp_areas;
return true;
}else{
return false;
}
})
}

this.setState({
options,
})
}

componentWillReceiveProps(nextProps) {
// Should be a controlled component.
if (this.state.default_value === null && 'value' in nextProps) {
let value = nextProps.value
if(!value){
value = []
}

if(value.length > 0){
const newValue = value.filter(function(item) {
return (!!item);
});
if(newValue.length !== value.length)
return;
}

this.props.initGeography(value);

let current_province = null;
let current_city = null;
let current_area = null;
if(nextProps.value.length === 3){
current_province = nextProps.value[0];
current_city = nextProps.value[1];
current_area = nextProps.value[2];
}

this.setState({
default_value: value,
current_province,
current_city,
current_area
})
}

if(this.state.default_value !== null)
this.updateConfig()
}

render() {
const loadData = (selectedOptions) => {
const targetOption = selectedOptions[selectedOptions.length - 1];
targetOption.loading = true;
if(targetOption.type == 'province'){
this.props.getCitys(targetOption.data);
this.setState({
current_province: targetOption.data.id,
})
}else if(targetOption.type == 'city'){
this.props.getAreas(targetOption.data);
this.setState({
current_city: targetOption.data.id,
})
}
targetOption.loading = false;
}

const onChange = (value, selectedOptions) => {
if(selectedOptions.length == 3 || selectedOptions.length == 0){
const onChange = this.props.onChange;
if(selectedOptions.length == 3){
this.setState({
current_province: selectedOptions[0].data.id,
current_city: selectedOptions[1].data.id,
current_area: selectedOptions[2].data.id,
})
if (onChange) {
onChange(selectedOptions.map(x=>x.data.id));
}
}else{
this.setState({
current_province: null,
current_city: null,
current_area: null,
})
if (onChange) {
onChange([]);
}
}
}
}

return (<span>
{(this.state.options.length > 0 && <Cascader
defaultValue={this.state.default_value}
onChange={onChange}
options={this.state.options}
loadData={loadData}
changeOnSelect
/>)}
</span>);
}
}


module.exports = {
Geography
};

可复用的model

import modelExtend from 'dva-model-extend'
import * as share from '../services/share'
import {model} from './common'

const geographyModel = modelExtend(model, {
state: {
provinces: [],
citys: [],
areas: [],
},

effects: {
* provinces({payload}, {call, put}) {
const {data} = yield call(share.provinces, {});
yield put({
type: 'updateState',
payload: {
provinces: data,
},
})
},

* citys({payload}, {call, put}) {
const {data} = yield call(share.citys, payload);
yield put({
type: 'updateState',
payload: {
citys: data,
},
})
},

* areas({payload}, {call, put}) {
const {data} = yield call(share.areas, payload);
yield put({
type: 'updateState',
payload: {
areas: data,
},
})
},

* selectProvince({payload}, {call, put}) {
yield put({
type: 'citys',
payload: payload['id'],
});

yield put({
type: 'updateState',
payload: {
current_province: payload,
},
});
},

* selectCity({payload}, {call, put}) {
yield put({
type: 'areas',
payload: payload['id'],
});

yield put({
type: 'updateState',
payload: {
current_city: payload,
},
});
},

* initGeography({payload}, {call, put}) {
let res = yield call(share.provinces, {});
const provinces = res.data

// 若没有默认值,只get省份列表
if(!payload || payload.length != 3){
yield put({
type: 'updateState',
payload: {
provinces,
citys: [],
areas: [],
},
})
return
}

const province_id = parseInt(payload[0]);
const city_id = parseInt(payload[1])
const area_id = parseInt(payload[2])

// 默认省份是否匹配
let current_province = null;
provinces.find(function(province){
if(province.id == province_id){
current_province = province;
return true
}else{
return false
}
})
if(!current_province) return

// 找到匹配的省份,get 市区
res = yield call(share.citys, current_province.id);
const citys = res.data

let current_city = null;
citys.find(function(city){
if(city.id == city_id){
current_city = city;
return true
}else{
return false
}
})
if(!current_city) return

res = yield call(share.areas, current_city.id);
const areas = res.data

let current_area = null;
areas.find(function(area){
if(area.id == area_id){
current_area = area;
return true
}else{
return false
}
})
if(!current_area) return

yield put({
type: 'updateState',
payload: {
provinces,
citys,
areas,
},
})
},
},

reducers: {
clearGeography(state,{ payload }) {
return {
...state,
provinces: [],
citys: [],
areas: [],
}
},
},

});

module.exports = {
geographyModel,
};

使用

const geographyProps = {
provinces: state.provinces,
citys: state.citys,
areas: state.areas,
initGeography(item) {
dispatch({
type: `${pageKey}/initGeography`,
payload: item
})
},
getCitys(item) {
dispatch({
type: `${pageKey}/selectProvince`,
payload: item
})
},
getAreas (item) {
dispatch({
type: `${pageKey}/selectCity`,
payload: item
})
},
};
export default modelExtend(geographyModel, {
<FormItem
label="位置"
required
{...formItemLayout}
>
{getFieldDecorator('geography', {
initialValue: state.editType === 'create' ? "" : [data.province_code,data.city_code,data.area_code],
rules: [
{
required: true,
}],
})(
<Geography {...geographyProps} />
)}
</FormItem>

gorm一些笔记

官网 https://github.com/jinzhu/gorm

数据表名

gorm默认将golang基于帕斯卡命名法的model的类名转换成下划线命名法格式作为数据表名字

若不一致,可以实现类方法TableName方法

func (StoreLite) TableName() string {
return "stores"
}

这个转换的方法有放开

// ToDBName convert string to db name
func ToDBName(name string) string

写个工具函数通过一个对象获取它的表名

import (
"github.com/jinzhu/gorm"
"github.com/jinzhu/inflection"
)
func DBName(obj interface{}) string {
t := reflect.TypeOf(obj)
_, ok := t.MethodByName("TableName")
if !ok { //没找到
if t.Kind() == reflect.Ptr {
t = t.Elem()
}
return inflection.Plural(gorm.ToDBName(t.Name()))
} else {
v := reflect.ValueOf(obj).MethodByName("TableName").Call([]reflect.Value{})
return v[0].Interface().(string)
}
}

通过外键关联查询时,会需要使用类名,参见http://jinzhu.me/gorm/associations.html#has-oneRelated

// User has one CreditCard, UserID is the foreign key
type User struct {
gorm.Model
CreditCard CreditCard
}

type CreditCard struct {
gorm.Model
UserID uint
Number string
}

var card CreditCard
db.Model(&user).Related(&card, "CreditCard")

go默认的反射的类名具有完整的包路径

func DBClassName(obj interface{}) string {
t := reflect.TypeOf(obj)
if t.Kind() != reflect.Struct {
t = t.Elem()
}
name := filepath.Ext(t.String())
return name[1:]
}

Select

gorm默认是select(*),有些情况可能会导致较多的不必要数据传输和性能损耗。解决方案是Select

db.Select("name, age").Find(&users)
//// SELECT name, age FROM users;

db.Select([]string{"name", "age"}).Find(&users)
//// SELECT name, age FROM users;

db.Table("users").Select("COALESCE(age,?)", 42).Rows()
//// SELECT COALESCE(age,'42') FROM users;

在代码中写死列名不是个好办法

// 过滤出需要查询的字段,若json:"-",则忽略之
// 背景: 默认情况下grom总是查询所有字段select(*),而实际情况并不需要那么多
func FieldsFilter(obj interface{}) []string {
dbName := DBName(obj)
value := reflect.TypeOf(obj)
if value.Kind() == reflect.Ptr {
value = value.Elem()
}

if value.Kind() != reflect.Struct {
panic("must be struct object")
}

var res []string
return fieldFilterImp(value, res, dbName)
}

func fieldFilterImp(t reflect.Type, res []string, dbName string) []string {
count := t.NumField()
for i := 0; i < count; i++ {
field := t.Field(i)
if field.Anonymous {
res = fieldFilterImp(field.Type, res, dbName)
continue
}

tag := field.Tag.Get("json")
name := parseTag(tag)
if name == "" {
name = field.Name
}
if name == "-" {
continue
}

tag = field.Tag.Get("field")
if tag == "-" {
continue
} else if tag != "" {
res = append(res, tag)
} else {
// 考虑到join之后可能有列的重名,所以加上db name
res = append(res, fmt.Sprintf("%s.%s", dbName, name))
}

}
return res
}

记录不存在的错误

当查询不存在时,同样返回错误,通过函数RecordNotFound判断是否属于这种情况

Count

gorm默认使用select count(*),相比select count(‘id’)效率低那么一点,可以优化

func GetObjectCount(obj interface{}, db *gorm.DB) (int, error) {
var cnt int
temp_db := db.Model(obj).Select("count(\"id\") as count").Count(&cnt)
if temp_db.Error != nil {
return 0, temp_db.Error
} else {
return cnt, nil
}
}

更新部分

可以使用Select/Omit来选择/排除更新的列

db.Model(&user).Select("name").Updates(map[string]interface{}{
"name": "hello", "age": 18, "actived": false})
//// UPDATE users SET name='hello',
// updated_at='2013-11-17 21:34:10' WHERE id=111;

db.Model(&user).Omit("name").Updates(map[string]interface{}{
"name": "hello", "age": 18, "actived": false})
//// UPDATE users SET age=18, actived=false,
// updated_at='2013-11-17 21:34:10' WHERE id=111;

同样利用上面的工具函数,可以过滤可更新的列 数据

批量更新

当需要一并插入多条数据,sql批量插入的写法比较省

INSERT INTO tbl_name
(a,b,c)
VALUES
(1,2,3),
(4,5,6),
(7,8,9);

不过gorm目前不支持https://github.com/jinzhu/gorm/issues/255

last_insert_id

gorm自动实现了last_insert_id,即便在事务模式下,后面的sql都可以直接取ID

tmpDB := gd.DB.Begin()
if err := service.CreateObject(apartment, tmpDB); err != nil {
tmpDB.Rollback()
util.ResponseDbError(http.StatusOK, c, err)
return
}

for k, v := range param.ImageUrls {
img := &model.ApartmentImage{
ApartmentID: apartment.ID,
Sequence: uint8(k + 1),
ImageUrl: v,
}
if err := service.CreateObject(img, tmpDB); err != nil {
tmpDB.Rollback()
util.ResponseDbError(http.StatusOK, c, err)
return
}
}

tmpDB.Commit()

软删除

type User struct {
gorm.Model
Birthday time.Time
}

// gorm源码
type Model struct {
ID uint `gorm:"primary_key"`
CreatedAt time.Time
UpdatedAt time.Time
DeletedAt *time.Time `sql:"index"`
}

grom确保自动更新CreatedAt/UpdatedAt,删除只会设置DeletedAt而不删除实际的数据

打印sql

打印gorm最终执行的sql对性能优化很有帮助

// LogMode set log mode, `true` for detailed logs,
// `false` for no log, default, will only print error logs
func (s *DB) LogMode(enable bool) *DB {
if enable {
s.logMode = 2
} else {
s.logMode = 1
}
return s
}

支持自定义对象

对于自定义类型变量,只需要实现database/sql.Scanner接口,参见https://github.com/jinzhu/gorm/issues/47

type FmtDate struct {
Tm *time.Time
}

const (
ctLayout = "2006-01-02 15:04:05"
cdLayout = "2006-01-02"
)

// sql.Scanner implementation to convert a time.Time column to a LocalDate
func (ct *FmtDate) Scan(value interface{}) error {
if tm, ok := value.(time.Time); ok {
ct.Tm = &tm
return nil
} else {
return fmt.Errorf("invalid time.Time format")
}
}

// sql/driver.Valuer implementation to go from LocalDate -> time.Time
func (ct FmtDate) Value() (driver.Value, error) {
if ct.Tm != nil {
return ct.Tm.Format(cdLayout), nil
} else {
return "", nil
}
}

// 为了支持json(序列化、反序列化),可以实现
func (ct *FmtDate) UnmarshalJSON(b []byte) (err error) {
s := strings.Trim(string(b), "\"")
if s == "null" {
return fmt.Errorf("invalid date format")
}
tm, err := time.Parse(cdLayout, s)
if err == nil {
ct.Tm = &tm
}
return err
}

func (ct FmtDate) MarshalJSON() ([]byte, error) {
if ct.Tm != nil {
return []byte(fmt.Sprintf("\"%s\"", ct.Tm.Format(cdLayout))), nil
} else {
return []byte("\"\""), nil
}
}

自动解析时间

https://stackoverflow.com/questions/29341590/go-parse-time-from-database

https://github.com/go-sql-driver/mysql#timetime-support

The default internal output type of MySQL DATE and DATETIME values is []byte which allows you to scan the value into a []byte, string or sql.RawBytes variable in your program.

However, many want to scan MySQL DATE and DATETIME values into time.Time variables, which is the logical opposite in Go to DATE and DATETIME in MySQL. You can do that by changing the internal output type from []byte to time.Time with the DSN parameter parseTime=true. You can set the default time.Time location with the loc DSN parameter.

Caution: As of Go 1.1, this makes time.Time the only variable type you can scan DATE and DATETIME values into. This breaks for example sql.RawBytes support.

Alternatively you can use the NullTime type as the scan destination, which works with both time.Time and string / []byte.

时区

gorm(应该是go-sql-driver/mysql)会使用utc时间,结果就是明明用9点写入数据库,但数据库里面存的是1点(东八区)

解决办法有几种

设置driver的loc为local

db, err := gorm.Open("mysql", 
"db:dbadmin@tcp(127.0.0.1:3306)/foo?charset=utf8&parseTime=true&loc=Local")

这样方案的好处是省事,不过在数据库中存当地时间,某些情况下(比如跨国家)会造成疑惑和bug,另一种变通的办法时,使用时间转成本地时间

type FmtTime struct {
Tm *time.Time
}

const (
ctLayout = "2006-01-02 15:04:05"
cdLayout = "2006-01-02"
)

func (ct *FmtTime) UnmarshalJSON(b []byte) (err error) {
s := strings.Trim(string(b), "\"")
if s == "null" {
return fmt.Errorf("invalid date format")
}
tm, err := time.Parse(ctLayout, s)
if err == nil {
ct.Tm = &tm
}
return err
}

func (ct *FmtTime) MarshalJSON() ([]byte, error) {
if ct.Tm != nil {
// 用本地时区输出
return []byte(fmt.Sprintf("\"%s\"", ct.Tm.In(time.Local).Format(ctLayout))), nil
} else {
return []byte("\"\""), nil
}
}

初始化

func getMysqlUrl(c *config.Config) string {
return fmt.Sprintf("%s:%s@tcp(%s:%d)/%s?charset=utf8&parseTime=true",
c.MysqlUser,
c.MysqlPassword,
c.MysqlHost,
c.MysqlPort,
c.MysqlDb)
}

func OpenDB(c *config.Config) (db *sql.DB, driver string) {
driver = "mysql"
var url string = getMysqlUrl(c)
db, err := sql.Open(driver, url)
if err != nil {
panic(err)
}
if driver == "mysql" {
// per issue https://github.com/go-sql-driver/mysql/issues/257
db.SetMaxIdleConns(0)
}

if err := pingDatabase(db); err != nil {
log.Fatal(err)
log.Fatalln("ping 数据库" + driver + "失败")
}
return
}

func OpenGorm(c *config.Config, driver string) (db *gorm.DB) {
url := getMysqlUrl(c)
db, err := gorm.Open(driver, url)
if err != nil {
log.Fatal(err)
}
return
}

func pingDatabase(db *sql.DB) (err error) {
for i := 0; i < 10; i++ {
err = db.Ping()
if err == nil {
return
}
log.Print("ping 数据库失败, 1s后重试")
time.Sleep(time.Second)
}
return
}

nginx编译

编译

http://nginx.org/download/下载nginx,最简单的用法

wget http://nginx.org/download/nginx-1.9.9.tar.gz
tar zvfx nginx-1.9.9.tar.gz
cd nginx-1.9.9.tar.gz
./configure
make
./obj/nginx

考虑到nginx有各种配置和各种可选的模块,通常需要定制configure的参数,configure –help查看所有的选项

可以【nginx -V】查看现有nginx的编译参数,若configure过程中缺少某些库,自行安装(ubuntu apt-get即可)

例如

./configure  --with-cc-opt='-g -O2 -fPIE -fstack-protector-strong \
-Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' \
--with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro \
-Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf \
--http-log-path=/var/log/nginx/access.log \
--error-log-path=/var/log/nginx/error.log \
--lock-path=/var/lock/nginx.lock \
--pid-path=/run/nginx.pid \
--http-client-body-temp-path=/var/lib/nginx/body \
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
--http-proxy-temp-path=/var/lib/nginx/proxy \
--http-scgi-temp-path=/var/lib/nginx/scgi \
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi \
--with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module \
--with-http_stub_status_module --with-http_realip_module \
--with-http_auth_request_module --with-http_addition_module \
--with-http_gunzip_module --with-http_gzip_static_module \
--with-http_v2_module --with-http_sub_module --with-stream \
--with-stream_ssl_module --with-threads

make之后,sudo ./objs/nginx 即可运行(background)模式

关闭nginx

#从容停止Nginx 
sudo kill -QUIT `pidof nginx | sed "s/ /\n/g" | sort | head -n 1`
#快速停止Nginx
sudo kill -TERM `pidof nginx | sed "s/ /\n/g" | sort | head -n 1`
#强制停止Nginx
sudo kill -9 `pidof nginx | sed "s/ /\n/g" | sort | head -n 1`

前台运行

nginx -g 'daemon off;'

编译第三方插件

随便挑一个https://www.nginx.com/resources/wiki/modules/fancy_index/

# 添加--add-module参数
./configure ** --add-module=/tmp/ngx-fancyindex
git clone https://github.com/aperezdc/ngx-fancyindex.git ngx-fancyindex
server {                                                                        
listen 8800;
root /home/king/;
fancyindex on;
fancyindex_exact_size off;
fancyindex_localtime on;
}

OpenResty

安装 https://openresty.org/cn/installation.html

可以安装预编译的二进制或者直接从源码编译

OpenResty以nginx的方式发布,只不过和官方的使用不同的模块。由于两者的配置等各种依赖隔离,只要确保OpenResty和Nginx不端口冲突,就可和谐共生

查看sudo make install 可以看到默认的安装路径

test -f '/usr/local/openresty/nginx/conf/mime.types' \
|| cp conf/mime.types '/usr/local/openresty/nginx/conf'
cp conf/mime.types '/usr/local/openresty/nginx/conf/mime.types.default'
test -f '/usr/local/openresty/nginx/conf/fastcgi_params' \
|| cp conf/fastcgi_params '/usr/local/openresty/nginx/conf'
cp conf/fastcgi_params \
'/usr/local/openresty/nginx/conf/fastcgi_params.default'
test -f '/usr/local/openresty/nginx/conf/fastcgi.conf' \
|| cp conf/fastcgi.conf '/usr/local/openresty/nginx/conf'
cp conf/fastcgi.conf '/usr/local/openresty/nginx/conf/fastcgi.conf.default'
test -f '/usr/local/openresty/nginx/conf/uwsgi_params' \
|| cp conf/uwsgi_params '/usr/local/openresty/nginx/conf'
cp conf/uwsgi_params \
'/usr/local/openresty/nginx/conf/uwsgi_params.default'
test -f '/usr/local/openresty/nginx/conf/scgi_params' \
|| cp conf/scgi_params '/usr/local/openresty/nginx/conf'
cp conf/scgi_params \
'/usr/local/openresty/nginx/conf/scgi_params.default'
# 主配置
test -f '/usr/local/openresty/nginx/conf/nginx.conf' \
|| cp conf/nginx.conf '/usr/local/openresty/nginx/conf/nginx.conf'
cp conf/nginx.conf '/usr/local/openresty/nginx/conf/nginx.conf.default'
test -d '/usr/local/openresty/nginx/logs' \
|| mkdir -p '/usr/local/openresty/nginx/logs'
test -d '/usr/local/openresty/nginx/logs' \
|| mkdir -p '/usr/local/openresty/nginx/logs'
test -d '/usr/local/openresty/nginx/html' \
|| cp -R docs/html '/usr/local/openresty/nginx'
test -d '/usr/local/openresty/nginx/logs' \
# 日志
|| mkdir -p '/usr/local/openresty/nginx/logs'
make[2]: Leaving directory '/tmp/openresty-1.13.6.1/build/nginx-1.13.6'
make[1]: Leaving directory '/tmp/openresty-1.13.6.1/build/nginx-1.13.6'
# 二进制
ln -sf /usr/local/openresty/nginx/sbin/nginx /usr/local/openresty/bin/openresty

网站统计中的数据收集原理及实现

配置

http {
# 创建名称为tick的日志格式(后面会引用)
log_format tick "$msec^A$remote_addr^A$u_domain^A$u_url^A$u_title^A$u_referrer^A$u_sh^A$u_sw^A$u_cd^A$u_lang^A$http_user_agent^A$u_utrace^A$u_account";

server {
listen 9088;
index index.html;

location / {
root html;
index index.html index.htm;
}

location /1.gif {
#伪装成gif文件
default_type image/gif;
#本身关闭access_log,通过subrequest记录log
access_log off;

access_by_lua "
-- 用户跟踪cookie名为__utrace
local uid = ngx.var.cookie___utrace
if not uid then
-- 如果没有则生成一个跟踪cookie,算法为md5(时间戳+IP+客户端信息)
uid = ngx.md5(ngx.now() .. ngx.var.remote_addr .. ngx.var.http_user_agent)
end
ngx.header['Set-Cookie'] = {'__utrace=' .. uid .. '; path=/'}
if ngx.var.arg_domain then
-- 通过subrequest到/i-log记录日志,将参数和用户跟踪cookie带过去
ngx.location.capture('/i-log?' .. ngx.var.args .. '&utrace=' .. uid)
end
";

#此请求不缓存
add_header Expires "Fri, 01 Jan 1980 00:00:00 GMT";
add_header Pragma "no-cache";
add_header Cache-Control "no-cache, max-age=0, must-revalidate";
#返回一个1×1的空gif图片
empty_gif;
}

location /i-log {
internal;

set_unescape_uri $u_domain $arg_domain;
set_unescape_uri $u_url $arg_url;
set_unescape_uri $u_title $arg_title;
set_unescape_uri $u_referrer $arg_referrer;
set_unescape_uri $u_sh $arg_sh;
set_unescape_uri $u_sw $arg_sw;
set_unescape_uri $u_cd $arg_cd;
set_unescape_uri $u_lang $arg_lang;
set_unescape_uri $u_utrace $arg_utrace;
set_unescape_uri $u_account $arg_account;

log_subrequest on;
#记录日志到ma.log,实际应用中最好加buffer,格式为tick
access_log /tmp/ma.log tick;

echo '';
}
}
}

前端代码
cat /usr/local/openresty/nginx/html/ma.js

(function () {
var params = {};
//Document对象数据
if(document) {
params.domain = document.domain || '';
params.url = document.URL || '';
params.title = document.title || '';
params.referrer = document.referrer || '';
}
//Window对象数据
if(window && window.screen) {
params.sh = window.screen.height || 0;
params.sw = window.screen.width || 0;
params.cd = window.screen.colorDepth || 0;
}
//navigator对象数据
if(navigator) {
params.lang = navigator.language || '';
}
//解析_maq配置
if(_maq) {
for(var i in _maq) {
switch(_maq[i][0]) {
case '_setAccount':
params.account = _maq[i][1];
break;
default:
break;
}
}
}
//拼接参数串
var args = '';
for(var i in params) {
if(args != '') {
args += '&';
}
args += i + '=' + encodeURIComponent(params[i]);
}

//通过Image对象请求后端脚本
var img = new Image(1, 1);
img.src = '/1.gif?' + args;
})();

cat /usr/local/openresty/nginx/html/index.html

<!DOCTYPE html>
<html>
<head>
<title>Welcome to OpenResty!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<script type="text/javascript">
var _maq = _maq || [];
_maq.push(['_setAccount', '网站标识']);

(function() {
var ma = document.createElement('script'); ma.type = 'text/javascript'; ma.async = true;
ma.src = '/ma.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ma, s);
})();
</script>
<h1>Welcome to OpenResty!</h1>
<p>If you see this page, the OpenResty web platform is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="https://openresty.org/">openresty.org</a>.<br/></p>

<p><em>Thank you for flying OpenResty.</em></p>
</body>
</html>

参考

  1. http://blog.codinglabs.org/articles/how-web-analytics-data-collection-system-work.html

Kelly源码剖析

Kelly是基于golang的一个简单的web框架

背景

作为web后端开发,标准的net/http非常高效灵活,足以适用非常多的场景,当然也有很多周边待补充,这就出现了各种web框架,甚至出现了替代默认的Http库的valyala/fasthttp

golang目前百花齐放,个人主要了解到的是两个项目

  1. beego: simple & powerful Go app framework
  2. gin-gonic/gin

beego没有实际用过,听说是大而全的项目,对开发者友好。不过由于了解甚少,草率评论并不合适,这里不作过多说明。

本着刨根问底的学习态度,最开始了解的是martini,后查证效率偏低(大量用到反射/reflect),所以就进一步学习了gin-gonic/gin

后者小巧灵活,学习成本低,并且提供了很多实用的补充,例如

  1. 路由和中间件核心框架,路由基于julienschmidt/httprouter
  2. gin.Context
  3. binding
  4. 校验,基于go-playground/validator.v9
  5. Http Request工具函数,获取param/path/form/header/cookie等
  6. Http Response工具函数,设置cookie,header,返回xml/json,返回template支持等
  7. 内建的几个常用中间件

martini/gin都包含非常多的中间件,两者迁移非常容易,参考

  1. https://github.com/codegangsta/martini-contrib
  2. https://github.com/gin-gonic/contrib
  3. https://github.com/gin-contrib

用久了,也发现gin也有一些问题

  1. 依赖还是偏多(虽然和很多库相比算较少的),就写个hello world都下载半天依赖
  2. 第三方middleware有的依赖gopkg.in的代码,另外一些依赖github.com的代码
  3. gin.Context对Golang标准库context不友好
  4. binding有一些问题,本人的优化版本在https://github.com/qjw/go-gin-binding
  5. 虽然middleware很多,但选择性太多,质量参差不齐,不好选择,另外太多的第三方依赖不如将大部分常用的集成到一起来的方便。

经过多方对比考察,认为urfave/negroni作为路由/中间件基础框架非常合适,(看看原型就知道他对context有多友好)所以折腾就开始了。

type Handler interface {
ServeHTTP(rw http.ResponseWriter, r *http.Request, next http.HandlerFunc)
}

经过综合评估,决定自己弄个类似于gin的框架

原则是尽量踏着巨人的肩膀,避免一些通用组件重复造轮子,聚焦于优秀智慧的集成

安装测试

安装kelly

go get github.com/qjw/kelly

运行sample

go get github.com/qjw/kelly/sample

具体参考https://github.com/qjw/kelly#运行sample

# 运行sample
king@king:~/tmp/gopath/bin$ ./sample
[negroni] listening on :9090

源码结构

  1. .(当前目录):核心代码
  2. binding:数据绑定支持,必需,自动安装
  3. render:响应输出支持,必须,自动安装,例如响应json/xml/html/text/模板/二进制,以及重定向等
  4. sample:测试代码
  5. sample-conf:测试代码
  6. toolkits:可选的辅助工具集,例如二维码/验证码/模板引擎,
  7. sessions:session/flash/认证/权限控制,可选,依赖redis
  8. middleware:各种中间件,可选
  9. middleware/swagger:swagger支持

路由

使用https://github.com/julienschmidt/httprouter

router需要支持以下特性

  1. 各种http方法
  2. Path变量
  3. 多级路由

httprouter并不原生多级路由,所以这里做了些扩展【留意代码中的注释,下同

type Router interface {
// 支持的http方法,支持链式调用
GET(string, ...HandlerFunc) Router
HEAD(string, ...HandlerFunc) Router
OPTIONS(string, ...HandlerFunc) Router
POST(string, ...HandlerFunc) Router
PUT(string, ...HandlerFunc) Router
PATCH(string, ...HandlerFunc) Router
DELETE(string, ...HandlerFunc) Router
}
type router struct {
// 共享的全局httprouter
rt *httprouter.Router
// 当前rouer路径
path string
// 绝对路径
absolutePath string
}

func (rt *router) GET(path string, handles ...HandlerFunc) Router {
// 留意rt.rt.GET作为参数传入
return rt.methodImp(rt.rt.GET, "GET", path, handles...)
}

func (rt *router) methodImp(
handle func(path string, handle httprouter.Handle),
method string,
path string,
handles ...HandlerFunc) Router {

// 增加计数
rt.endpoints = append(rt.endpoints, &endpoint{
method: method,
path: path,
handles: handles,
endPointRegisterCB: func() {
// 在调用传入的handle函数前,已经加上了rt.absolutePath
handle(rt.absolutePath+path, rt.wrapHandle(handles...))
},
})
return rt
}

kelly的router都共享一个全局的httprouter对象,并且各自保存自己的所处的路径(path),在绑定特定的http请求时,自动加上自己的前缀,再绑定到原生的httprouter上去

Callback和Context

golang自带的net/http callback为

type Handler interface {
ServeHTTP(ResponseWriter, *Request)
}

type HandlerFunc func(ResponseWriter, *Request)
func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
f(w, r)
}

Context则是一个输入(Request),一个输出/响应(ResponseWriter),这种方法有很好的灵活性,不过易用性稍弱

kelly借鉴gin的做法

type HandlerFunc func(c *Context)

func (f HandlerFunc) ServeHTTP(w http.ResponseWriter, r *http.Request) {
f(newContext(w, r, nil))
}

其中最关键的就是这个Context,因为web编程大部分工作(刨去业务代码),最重要的还是解析输入,构建输出,这些都是基于http请求,这就是Context的抽象

type Context struct {
http.ResponseWriter
r *http.Request

// 下一个处理逻辑,用于middleware
next http.HandlerFunc

// 用于支持设置context数据
dataContext
// render
renderOp
// request
request
// binder
binder
}

可以看到Context除了包装了http.Request和http.ResponseWriter之外,还有一些

  1. dataContext :支持请求context,用于在中间件链传递数据。参见源码目录render
  2. renderOp :格式化输出支持(例如xml/json等)
  3. binder : 数据绑定支持,参见源码目录binding
  4. request :输入(获取参数)支持

binder和request在于,前者是自动将http请求的输入绑定到一个golang struct,并且依据规则进行校验,提高开发效率。后者则属于一些工具函数,例如获取某个param/path/form参数,或者获取某个cookie。

type binder interface {
// 绑定一个对象,根据Content-type自动判断类型
Bind(interface{}) (error, []string)
// 绑定json,从body取数据
BindJson(interface{}) (error, []string)
// 绑定xml,从body取数据
BindXml(interface{}) (error, []string)
// 绑定form,从body/query取数据
BindForm(interface{}) (error, []string)
// 绑定path变量
BindPath(interface{}) (error, []string)

GetBindParameter() interface{}
GetBindJsonParameter() interface{}
GetBindXmlParameter() interface{}
GetBindFormParameter() interface{}
GetBindPathParameter() interface{}
}
type request interface {
// 根据key获取cookie值
GetCookie(string) (string, error)
// 根据key获取cookie值,若不存在,则返回默认值
GetDefaultCookie(string, string) string
// 根据key获取cookie值,若不存在,则panic
MustGetCookie(string) string

// 根据key获取header值
GetHeader(string) (string, error)
// 根据key获取header值,若不存在,则返回默认值
GetDefaultHeader(string, string) string
// 根据key获取header值,若不存在,则panic
MustGetHeader(string) string
// Content-Type
ContentType() string

// 根据key获取PATH变量值
GetPathVarible(string) (string, error)
// 根据key获取PATH变量值,若不存在,则panic
MustGetPathVarible(string) string

// 根据key获取QUERY变量值,可能包含多个(http://127.0.0.1:9090/path/abc?abc=bbb&abc=aaa)
GetMultiQueryVarible(string) ([]string, error)
// 根据key获取QUERY变量值,仅返回第一个
GetQueryVarible(string) (string, error)
// 根据key获取QUERY变量值,仅返回第一个,若不存在,则返回默认值
GetDefaultQueryVarible(string, string) string
// 根据key获取QUERY变量值,仅返回第一个,若不存在,则panic
MustGetQueryVarible(string) string

// 根据key获取FORM变量值,可能get可能包含多个
GetMultiFormVarible(string) ([]string, error)
// 根据key获取FORM变量值,仅返回第一个
GetFormVarible(string) (string, error)
// 根据key获取FORM变量值,仅返回第一个,若不存在,则返回默认值
GetDefaultFormVarible(string, string) string
// 根据key获取FORM变量值,仅返回第一个,若不存在,则panic
MustGetFormVarible(string) string

// @ref http.Request.ParseMultipartForm
ParseMultipartForm() error
// 获取(上传的)文件信息
GetFileVarible(string) (multipart.File, *multipart.FileHeader, error)
MustGetFileVarible(string) (multipart.File, *multipart.FileHeader)
}

render处理常见的xml/json等之外,还有比如设置cookie/header等操作,完成的功能列表如下

type renderOp interface {
// 返回紧凑的json
WriteJson(int, interface{})
// 返回xml
WriteXml(int, interface{})
// 返回html
WriteHtml(int, string)
// 返回模板html
WriteTemplateHtml(int, *template.Template, interface{})
// 返回格式化的json
WriteIndentedJson(int, interface{})
// 返回文本
WriteString(int, string, ...interface{})
// 返回二进制数据
WriteData(int, string, []byte)
// 返回重定向
Redirect(int, string)
// 设置header
SetHeader(string, string)
// 设置cookie
SetCookie(string, string, int, string, string, bool, bool)

Abort(int, string)
ResponseStatusOK()
ResponseStatusBadRequest(error)
ResponseStatusUnauthorized(error)
ResponseStatusForbidden(error)
ResponseStatusNotFound(error)
ResponseStatusInternalServerError(error)
}

这里的WriteTemplateHtml使用golang内置的模板引擎,由于功能有限,在【toolkits/template】中基于第三方引擎实现更为强大的功能

context则比较简单,只是一个数据的get/set

type dataContext interface {
Set(interface{}, interface{}) dataContext
Get(interface{}) interface{}
MustGet(interface{}) interface{}
}

中间件框架

好的中间件框架是一个web框架灵活性的重要标志,目前主流的框架大部分都支持中间件扩展,并且有丰富的第三方中间件。为了方便移植其他框架的中间件,一般都容易兼容标准的net/http接口。

标准的net/http默认不支持读写自定义数据,这很大程度影响中间件的灵活性,当然可以用标准库context进行扩充,由于需要替换原有的reqeust,对程序结构影响很大,参见http://www.flysnow.org/2017/07/29/go-classic-libs-gorilla-context.html#新的替代者。经过调研,选用https://github.com/urfave/negroni

negroni原型如下

type HandlerFunc func(rw http.ResponseWriter, r *http.Request, next http.HandlerFunc)

func (h HandlerFunc) ServeHTTP(rw http.ResponseWriter, r *http.Request, next http.HandlerFunc) {
h(rw, r, next)
}

negroni中间件默认不会传递调用,而是需要手动触发调用参数next,所以当使用context添加data并替换了http.Request对象时,处理起来就很轻松和自然

回调调用链

kelly的入口在kelly.Run


type kellyImp struct {
*router
n *negroni.Negroni
}

func (k *kellyImp) Run(addr ...string) {
// 运行negroni.Negroni.Run
k.n.Run(addr...)
}

func newImp(n *negroni.Negroni, handlers ...HandlerFunc) Kelly {
// 创建negroni.Negroni回调
rt := newRouterImp(handlers...)
ky := &kellyImp{
router: rt,
n: n,
}
ky.n = n

// negroni触发之后,会进入rt
n.UseHandler(rt)
return ky
}

当kelly收到http请求,先触发negroni.Negroni的回调,这个回调在router定义

func newRouterImp(handlers ...HandlerFunc) *router {
// 创建全局的httprouter对象
httpRt := httprouter.New()
// 创建根router对象
rt := &router{
rt: httpRt,
path: "", // 根router路径是空的
absolutePath: "",
dataContext: newMapContext(),
}
return rt
}

而router又将请求转发到了全局的httprouter对象,参见下面代码

func (rt *router) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
rt.rt.ServeHTTP(rw, r)
}

最终就触发了我们在注册http方法时设置到httprouter的回调中

中间件

  1. 有两种方式设置中间件,在创建router时,传入
  2. 创建之后,调用Use
// 创建根router
func NewClassic(handlers ...HandlerFunc) Kelly {
return newImp(negroni.Classic(), handlers...)
}

// 创建根router
func New(handlers ...HandlerFunc) Kelly {
return newImp(negroni.New(negroni.NewRecovery()), handlers...)
}

type Router interface {
// 新建子router
Group(string, ...HandlerFunc) Router

// 动态插入中间件
Use(...HandlerFunc) Router
}

由于httprouter并没有对中间件的支持,所以需要继续作转换

最外层的注册,使用了kelly的回调原型

type router struct {
// 中间件
middlewares []HandlerFunc

// 所有的子Group
groups []*router

// 父Group
parent *router

endpoints []*endpoint
}

func (rt *router) GET(path string, handles ...HandlerFunc) Router {
return rt.methodImp(rt.rt.GET, "GET", path, handles...)
}

留意结构中的endPointRegisterCB成员rt.wrapHandle(handles…)

func (rt *router) methodImp(
handle func(path string, handle httprouter.Handle),
method string,
path string,
handles ...HandlerFunc) Router {

// 将注册信息加入到endpoints成员中,
rt.endpoints = append(rt.endpoints, &endpoint{
method: method,
path: path,
handles: handles,
endPointRegisterCB: func() {
// 注册回调,
handle(rt.absolutePath+path, rt.wrapHandle(handles...))
},
})
return rt
}

在kelly进入主循环之前,会完成调用。留意doBeforeRun

func (k *kellyImp) Run(addr ...string) {
if k.n == nil {
panic("invalid kelly")
}
k.router.doBeforeRun()
k.n.Run(addr...)
}

func (rt *router) doBeforeRun() {
// 先注册自己的endpoints
for _, v := range rt.endpoints {
v.run()
}
// 先注册儿子们的endpoints
for _, v := range rt.groups {
v.doBeforeRun()
}
}
func (this *endpoint) run() {
if DebugFlag && this.endPointRegisterCB == nil {
panic("invalid endpoint")
}
// 直接调用创建时的函数,并且清空,避免重复调用
this.endPointRegisterCB()
this.endPointRegisterCB = nil
}

之所以通过这种方式,在最后一步统一注册,是为了支持创建router之后使用Use方法动态添加中间件的情形

Callback转换

在endPointRegisterCB中,注册的回调需要原型【httprouter.Handle】,而我们传入的是【kelly.HandlerFunc数组】,通过下面的函数转换

func (rt *router) wrapHandle(handles ...HandlerFunc) httprouter.Handle {
// 创建一个negroni实例
tmpHandle := negroni.New()
// 将父router的中间件(【kelly.HandlerFunc数组】)注册到negroni
rt.wrapParentHandle(tmpHandle)

// 注册当前router的中间件到negroni
for _, v := range rt.middlewares {
tmpHandle.UseFunc(wrapHandlerFunc(v))
}

// 注册特定方法的中间件到negroni
for _, v := range handles {
tmpHandle.UseFunc(wrapHandlerFunc(v))
}

// 返回一个httprouter的回调
return func(wr http.ResponseWriter, r *http.Request, params httprouter.Params) {
// 将请求转到negroni
tmpHandle.ServeHTTP(wr, r)
}
}

func (rt *router) wrapParentHandle(n *negroni.Negroni) {
if rt.parent != nil {
// 让父router优先注册自己的中间件
rt.parent.wrapParentHandle(n)
// 注册parent的中间件
for _, v := range rt.parent.middlewares {
n.UseFunc(wrapHandlerFunc(v))
}
}
}

大体的思路就是每次注册http方法,就生成一个negroni对象,并且注册祖宗十八代的中间件、自己的中间件和自己的处理函数,然后转成httprouter的回调注册。

而negroni包装kelly.HandlerFunc回调时,同样需要一层转换

func wrapHandlerFunc(f HandlerFunc) negroni.HandlerFunc {
return func(rw http.ResponseWriter, r *http.Request, next http.HandlerFunc) {
f(newContext(rw, r, next))
}
}

留意这个next,为了支持中间件链继续运行,需要保存这个next

func newContext(w http.ResponseWriter, r *http.Request, next http.HandlerFunc) *Context {
c := &Context{
next: next,
}
return c
}
func (c *Context) InvokeNext() {
if c.next != nil {
c.next.ServeHTTP(c, c.Request())
} else {
panic("invalid invoke next")
}
}

性能

一个请求进来,先走negroni,转发到httprouter,httprouter根据路由规则找到对应的negroni回调,然后触发每一个中间件回调,最后到业务回调。这个过程中,需要将negroni参数转换成kelly.HandlerFunc

这整个过程中存在

  1. 额外的内存消耗,每个请求都会有一个negroni实例
  2. 函数调用栈过长带来的cpu消耗
  3. 每个中间件/回调都伴随这一个kelly.Context对象的创建(创建本身比较简单)

Path变量

path变量 httprouter支持,具体存储在回调的第三个参数中

type Handle func(http.ResponseWriter, *http.Request, Params)

type Param struct {
Key string
Value string
}

type Params []Param

回到函数wrapHandle,留意mapContextFilter

func (rt *router) wrapHandle(handles ...HandlerFunc) httprouter.Handle {
return func(wr http.ResponseWriter, r *http.Request, params httprouter.Params) {
r = mapContextFilter(wr, r, params)
tmpHandle.ServeHTTP(wr, r)
}
}

实际上就是将这个params参数通过标准库context存入request对象中

type contextMap map[interface{}]interface{}
func mapContextFilter(_ http.ResponseWriter, r *http.Request, params httprouter.Params) *http.Request{
contextMap := contextMap{
pathParamID: params,
}
return contextSet(r, contextKey, contextMap)
}
func contextSet(r *http.Request, key, value interface{}) *http.Request {
ctx := context.WithValue(r.Context(), key, value)
return r.WithContext(ctx)
}

取变量

func contextMustGet(r *http.Request, key interface{}) interface{} {
v := r.Context().Value(key)
if v == nil {
panic(fmt.Errorf("get context value fail by '%v'", key))
}
return v
}

func getPathParams(r *http.Request) httprouter.Params{
datas := contextMustGet(r, contextKey).(contextMap)
// 这个pathParamID是一个全局常量
return datas[pathParamID].(httprouter.Params)
}
func (r requestImp) GetPathVarible(name string) (string, error) {
params := getPathParams(r.Request)
val := params.ByName(name)
if len(val) > 0 {
return val, nil
} else {
return val, fmt.Errorf("can not get path varibel by '%v'", name)
}
}

Context数据

由于kelly.Context在整个调用链并非连续(而是每个negroni调用动态创建的),所以不能简单地在里面加一个map之类的成员

func newContext(w http.ResponseWriter, r *http.Request, next http.HandlerFunc) *Context {
c := &Context{
dataContext: newMapHttpContext(r),
}
return c
}

所以和path变量一样,同样将一个map对象绑定到request对象

type mapHttpContext struct {
r *http.Request
}

func (c *mapHttpContext) Set(key, value interface{}) dataContext {
datas := contextMustGet(c.r, contextKey).(contextMap)
datas[key] = value
return c
}

func (c mapHttpContext) Get(key interface{}) interface{} {
datas := contextMustGet(c.r, contextKey).(contextMap)
if data, ok := datas[key]; ok {
return data
} else {
return nil
}
}

func newMapHttpContext(r *http.Request) dataContext {
c := &mapHttpContext{
r: r,
}
return c
}

注解

中间件可以在每个请求之前做些处理,甚至拦截,注解则在每次注册Http请求时做些处理,比如swagger就依赖于这种场景

type Router interface {
// 添加全局的 注解 函数。该router下面和子(孙)router下面的endpoint注册都会被触发
GlobalAnnotation(handles ...AnnotationHandlerFunc) Router

// 添加临时 注解 函数,只对使用返回的AnnotationRouter对象进行注册的endpoint有效
Annotation(handles ...AnnotationHandlerFunc) AnnotationRouter

// 用于支持设置context数据
dataContext
}

type AnnotationHandlerFunc func(c *AnnotationContext)

type AnnotationContext struct {
r Router
method string
path string
// 不包含中间件
handles []HandlerFunc
}

注册filter

func (rt *router) GlobalAnnotation(handles ...AnnotationHandlerFunc) (r Router) {
r = rt
if len(rt.epMiddlewares) == 0 {
rt.epMiddlewares = make([]AnnotationHandlerFunc, len(handles))
copy(rt.epMiddlewares, handles)
} else {
for _, item := range handles {
rt.epMiddlewares = append(rt.epMiddlewares, item)
}
}
return
}

只是简单地将他存到成员epMiddlewares中

type router struct {
// endpoint钩子函数
epMiddlewares []AnnotationHandlerFunc

// 被子类覆盖的方法,
overiteInvokeAnnotation func(c *AnnotationContext)
}

触发

创建router时,会指定rt。在doBeforeRun时,会完成注入

func newRouterImp(handlers ...HandlerFunc) *router {
rt := &router{}
rt.overiteInvokeAnnotation = rt.invokeAnnotation
return rt
}

func (rt *router) invokeParentAnnotation(c *AnnotationContext) {
if rt.parent != nil {
rt.parent.invokeParentAnnotation(c)
for _, item := range rt.parent.epMiddlewares {
item(c)
}
}
}

func (rt *router) invokeAnnotation(c *AnnotationContext) {
rt.invokeParentAnnotation(c)

// 执行全局的ep 过滤器
for _, item := range rt.epMiddlewares {
item(c)
}
}

再回到注册http请求的函数methodImp。留意函数变量f

func (rt *router) methodImp(
handle func(path string, handle httprouter.Handle),
method string,
path string,
handles ...HandlerFunc) Router {

f := rt.overiteInvokeAnnotation

// 增加计数
rt.endpoints = append(rt.endpoints, &endpoint{
endPointRegisterCB: func() {
// 注册到httprouter
handle(rt.absolutePath+path, rt.wrapHandle(handles...))
// 调用注解
f(&AnnotationContext{
r: rt,
method: method,
path: path,
handles: handles,
})
},
})
return rt
}

临时Filter

使用GlobalAnnotation注册filter会应用到当前router和他的子router,临时filter需要使用kelly.Router.Annotation。这个函数返回的是一个AnnotationRouter对象。

这个对象有自己的AnnotationHandlerFunc数组,所以不会影响其他的请求

func newAnnotationRouter(r *router, handles ...AnnotationHandlerFunc) AnnotationRouter {
return &annotationRouter{
router: r,
middlewares: handles,
}
}

type annotationRouter struct {
*router
// endpoint钩子函数
middlewares []AnnotationHandlerFunc
}

同时为了在触发filter时,一并触发自己的filter,需要重写

func (r *annotationRouter) doMethod(
f func(path string, handles ...HandlerFunc) Router,
path string,
handles ...HandlerFunc,
) Router {
old := r.router.overiteInvokeAnnotation
// 重写overiteInvokeAnnotation,并且函数退出自动复原
r.router.overiteInvokeAnnotation = r.invokeAnnotation
defer func() {
r.router.overiteInvokeAnnotation = old
}()
return f(path, handles...)
}

由于go并非真正的继承,而只是简单的组合,所以这里的多态实现有些另类

注解使用

必须使用链式调用,或者保存返回的临时对象

router := r.Group("/swagger"
).GlobalAnnotation(swagger.SetGlobalParam(&swagger.StructParam{
Tags: []string{"API接口"},
})).OPTIONS("/*path", func(c *kelly.Context) {
c.ResponseStatusOK()
})

router.Annotation(swagger.Swagger(&swagger.StructParam{
ResponseData: &swagger.SuccessResp{},
FormData: &swaggerParam{},
Summary: "api1",
})).POST("/api1", func(c *kelly.Context) {
c.ResponseStatusOK()
})

Golang 调试

编译选项

传递-gcflags “-N -l” 参数,这样可以忽略Go内部做的一些优化,聚合变量和函数等优化,这样对于GDB调试来说非常困难,所以在编译的时候加入这两个参数避免这些优化。

  1. 对于单个go文件的,直接 【go build -gcflags “-N -l” main.go】
  2. 对于基于package的,使用包路径 【go build -gcflags “-N -l” github.com/qjw/kelly/sample】,在当前目录直接.即可【go build -gcflags “-N -l” .】

GDB

处理下断点时,注意函数的名称,其他都和C一致

以main函数为例,不能直接b main,这虽然会生效,但是实际上会陷入c库的main,而不是golang的main,务必使用包名+函数名

king@king:/go/src/github.com/qjw/kelly/sample$ gdb sample 
(gdb) b main.main
Breakpoint 1 at 0x8bfe70: file /go/src/github.com/qjw/kelly/sample/main.go, line 42.
(gdb) l
28 Password: "",
29 DB: 3,
30 })
31 if err := redisClient.Ping().Err(); err != nil {
32 log.Fatal("failed to connect redis")
33 }

Delve

安装指引 https://github.com/derekparker/delve/blob/master/Documentation/installation/linux/install.md,简单地就可以

go get github.com/derekparker/delve/cmd/dlv

最终debug程序就是dlv,目前Gogland等很多IDE的调试器就使用dlv。

king@king:/go/src/github.com/qjw/kelly/sample$ dlv exec ./sample 
Type 'help' for list of commands.
(dlv) b main.main
Breakpoint 1 set at 0x8bfe8b for main.main() ./main.go:42
(dlv) continue
> main.main() ./main.go:42 (hits goroutine(1):1 total:1) (PC: 0x8bfe8b)
37: log.Print(err)
38: }
39: return store
40: }
41:
=> 42: func main() {
43: store := initStore()

自动从源码编译

king@king:/go/src/github.com/qjw/kelly/sample$ dlv debug .
Type 'help' for list of commands.
(dlv) b main.main
Breakpoint 1 set at 0xa1d2db for main.main() ./main.go:42
(dlv) continue
> main.main() ./main.go:42 (hits goroutine(1):1 total:1) (PC: 0xa1d2db)
37: log.Print(err)
38: }
39: return store
40: }
41:
=> 42: func main() {

Golang 可选类型

对于struct,有些字段可能是空值,有些字段不存在(这是两种含义),若对于不允许空值的场景,空值可以表示不存在的含义。

  1. golang声明之后会赋一个默认值(区别C/C++)
  2. 用万能的指针可以区分空值和不存在,不过存在Null Pointer Exception的问题

以下内容参考http://www.markphelps.me/2017/08/20/optional-types-with-go-generate.html

既然不用指针,就必须有一个标志表示是否存在,对象定义如下

package optional

type String struct {
string string
present bool
}

func EmptyString() String {
return String{}
}

func OfString(s string) String {
return String{string: s, present: true}
}

func (o *String) Set(s string) {
o.string = s
o.present = true
}

func (o *String) Get() string {
return o.string
}

func (o *String) Present() bool {
return o.present
}

由于golang没有语言层面的模板(类比C++的template),而type却有很多种,所以需要有一种机制能快速给一个对象生成上述的代码。

利用go template机制和go generate比较容易实现

模板定义

package {{ .PackageName }}

// {{ .OutputName }} is an optional {{ .TypeName }}
type {{ .OutputName }} struct {
{{ .TypeName | unexport }} {{ .TypeName }}
present bool
}

// Empty{{ .OutputName | title }} returns an empty {{ .PackageName }}.{{ .OutputName }}
func Empty{{ .OutputName | title }}() {{ .OutputName }} {
return {{ .OutputName }}{}
}

// Of{{ .TypeName | title }} creates a {{ .PackageName }}.{{ .OutputName }} from a {{ .TypeName }}
func Of{{ .TypeName | title }}({{ .TypeName | first }} {{ .TypeName }}) {{ .OutputName }} {
return {{ .OutputName }}{ {{ .TypeName | unexport }}: {{ .TypeName | first }}, present: true}
}

// Set sets the {{ .TypeName }} value
func (o *{{ .OutputName }}) Set({{ .TypeName | first }} {{ .TypeName }}) {
o.{{ .TypeName | unexport }} = {{ .TypeName | first }}
o.present = true
}

// Get returns the {{ .TypeName }} value
func (o *{{ .OutputName }}) Get() {{ .TypeName }} {
return o.{{ .TypeName | unexport }}
}

// Present returns whether or not the value is present
func (o *{{ .OutputName }}) Present() bool {
return o.present
}

代码定义

import (
"time"
"strings"
"html/template"
"bytes"
"io/ioutil"
"log"
)

type data struct {
Timestamp time.Time
PackageName string
TypeName string
OutputName string
}

var (
funcMap = template.FuncMap{
"title": strings.Title,
"first": func(s string) string {
return strings.ToLower(string(s[0]))
},
"unexport": func(s string) string {
return strings.ToLower(string(s[0])) + string(s[1:])
},
}
)

func main() {
temp,err := ioutil.ReadFile("src/github.com/qjw/test/a.tpl")
if err != nil{
log.Panic(err)
return
}


t := template.Must(template.New("").Funcs(funcMap).Parse(string(temp)))


var buf bytes.Buffer
err = t.Execute(&buf, data{
Timestamp:time.Now(),
PackageName: "main",
TypeName: "int",
OutputName: "Int",

})
if err != nil {
log.Panic(err)
return
}

err = ioutil.WriteFile("src/github.com/qjw/test/a_gen.go", buf.Bytes(), 0644)
if err != nil {
log.Fatalf("writing output: %s", err)
}
}

也可以将其作为一个command,然后用go generate批量生成类型代码,作者的代码在https://github.com/markphelps/optional/blob/master/cmd/optional/main.go

Golang第三方工具

360EntSecGroup-Skylar/goreporter

A Golang tool that does static analysis, unit testing, code review and generate code quality report.

go get -u github.com/360EntSecGroup-Skylar/goreporter

安装之后,为了支持生成图表,需要

sudo apt-get install graphviz

运行

由于360EntSecGroup-Skylar/goreporter 不会自动查找和排除vendor目录,所以这里取巧,将vendor/src做一个软连接到vendor目录

ln -s github.com/qjw/git-notify/vendor/ github.com/qjw/git-notify/vendor/src
# html结果文件在当前目录,可以参数另行指定
./goreporter -p ../src/github.com/qjw/git-notify/ -e ../src/github.com/qjw/git-notify/vendor/ -f html

感觉没有什么比较有价值的东西,生成的报表界面倒是挺好看

stringer

根据一个( unsigned )int的别名生成fmt.Stringer接口的方法

Stringer is a tool to automate the creation of methods that satisfy the fmt.Stringer interface. Given the name of a (signed or unsigned) integer type T that has constants defined, stringer will create a new self-contained Go source file implementing

go get golang.org/x/tools/cmd/stringer

安装完了会编译出一个stringer命令,在$GOPATH/bin

待生成方法的int别名Pill

package painkiller

type Pill int

const (
Placebo Pill = iota
Aspirin
Ibuprofen
Paracetamol
Acetaminophen = Paracetamol
)

运行命令

stringer -type=Pill

结果

// Code generated by "stringer -type=Pill"; DO NOT EDIT.

package painkiller

import "fmt"

const _Pill_name = "PlaceboAspirinIbuprofenParacetamol"

var _Pill_index = [...]uint8{0, 7, 14, 23, 34}

func (i Pill) String() string {
if i < 0 || i >= Pill(len(_Pill_index)-1) {
return fmt.Sprintf("Pill(%d)", i)
}
return _Pill_name[_Pill_index[i]:_Pill_index[i+1]]
}

go-callvis

生成调用图

Purpose of this tool is to provide a visual overview of your program by using the data from call graph and its relations with packages and types. This is especially useful in larger projects where the complexity of the code rises or when you are just simply trying to understand code structure of somebody else.

安装

go get -u github.com/TrueFurby/go-callvis
cd $GOPATH/src/github.com/TrueFurby/go-callvis && make

使用

必须包含main package

go-callvis github.com/qjw/kelly/sample | dot -Tpng -o output.png

生成的png图片可能很大

gometalinter

执行各种静态检查,支持各种第三方的lint tools

Concurrently run Go lint tools and normalise their output

go get -u gopkg.in/alecthomas/gometalinter.v1
gometalinter.v1 --install
gometalinter.v1 --vendor --skip=vendor...

depth

查看依赖树

king@king:~/code/go/src/bb$ depth github.com/qjw/kelly
github.com/qjw/kelly
├ github.com/qjw/kelly/binding
└ gopkg.in/go-playground/validator.v9
└ github.com/go-playground/universal-translator
└ github.com/go-playground/locales
└ github.com/go-playground/locales/currency
├ github.com/qjw/kelly/render
├ github.com/julienschmidt/httprouter
├ github.com/urfave/negroni

Flask sqlalchemy调优总结

直接用tcpdump监听数据库端口(mysql为例)可以打印所有的sql语句。(注意修改网卡的名称)

king@king:/tmp$ sudo tcpdump -i lo -s 0 -l -w - dst port 3306 | strings | perl -e '
while(<>) { chomp; next if /^[^ ]+[ ]*$/;
if(/^(SELECT|UPDATE|DELETE|INSERT|SET|COMMIT|ROLLBACK|CREATE|DROP|ALTER)/i) {
if (defined $q) { print "$q\n"; }
$q=$_;
} else {
$_ =~ s/^[ \t]+//; $q.=" $_";
}
}'

由于stdout缓存,会有大概率最后几条出不来,(继续执行sql语句就会把这一次没出来的挤出来)

sqlalchemy官网可以设置一些时间,用于监听任务执行的时间。注意

  1. before_cursor_execute
  2. after_cursor_execute
from sqlalchemy import event
from sqlalchemy.engine import Engine
import time
import logging

logging.basicConfig()
logger = logging.getLogger("myapp.sqltime")
logger.setLevel(logging.DEBUG)

@event.listens_for(Engine, "before_cursor_execute")
def before_cursor_execute(conn, cursor, statement,
parameters, context, executemany):
conn.info.setdefault('query_start_time', []).append(time.time())
logger.debug("Start Query: %s", statement)

@event.listens_for(Engine, "after_cursor_execute")
def after_cursor_execute(conn, cursor, statement,
parameters, context, executemany):
total = time.time() - conn.info['query_start_time'].pop(-1)
logger.debug("Query Complete!")
logger.debug("Total Time: %f", total)

完整的事件文档参考

  1. http://docs.sqlalchemy.org/en/latest/core/events.html
  2. http://docs.sqlalchemy.org/en/latest/orm/events.html
  3. http://docs.sqlalchemy.org/en/latest/core/event.html#event-reference

使用工具line_profiler分析调用时间,使用非常简单,在需要分析的函数加上【@prifile】注解,

若使用ide,会提示注解找不到符号,不用管他

@profile
def run():
a = [1]*100
b = [x*x*x for x in a ]
c = [x for x in b]
d = c*2
run()

安装line_profiler会一并安装一个kernprof.py脚本,需要这个脚本运行我们的程序才会不出错

在程序退出后,-v参数会指示程序将结果输出到stdout,例如

Total time: 8.55619 s
File: /home/king/bug/venv/lib/python3.5/site-packages/sqlalchemy/orm/query.py
Function: __iter__ at line 2733

Line # Hits Time Per Hit % Time Line Contents
==============================================================
2733 @profile
2734 def __iter__(self):
2735 2176 1140139 524.0 13.3 context = self._compile_context()
2736 2176 3682 1.7 0.0 context.statement.use_labels = True
2737 2176 2284 1.0 0.0 if self._autoflush and not self._populate_existing:
2738 2176 28075 12.9 0.3 self.session._autoflush()
2739 2176 7382008 3392.5 86.3 return self._execute_and_instances(context)

参考https://www.oschina.net/translate/python-performance-analysis?print

也可以用cProfile来分析,参考

只需要在运行时,加上【-m cProfile】python选项

python -m cProfile /home/king/code/bug/main.py

然后在需要作分析测试的地方

cProfile.run('requests.get("http://tech.glowing.com")')

一种更简单的办法

import cProfile
import io
import pstats
import contextlib

@contextlib.contextmanager
def profiled(limit=20):
pr = cProfile.Profile()
pr.enable()
yield
pr.disable()
s = io.StringIO()
ps = pstats.Stats(pr, stream=s).sort_stats('cumulative')
ps.print_stats(limit)
# uncomment this to see who's calling what
# ps.print_callers()
print(s.getvalue())

def Test(cars):
with profiled(limit=30):
for num in range(1, 20):
for car in cars:
car.test()

参考

  1. http://docs.sqlalchemy.org/en/latest/faq/performance.html
  2. http://tech.glowing.com/cn/python-profiling/
  3. http://ju.outofmemory.cn/entry/284150