erlang ets性能优化

假如 ets 里存的数据格式如下

1
{{user_id, type}, pid}

ordered_set, keypos = 1

要用上 ordered_set 的前缀匹配高性能, match specification 需要像下面这样写

1
[{{{user_id, :_}, :"$1"}, [], [:"$1"]}]

这样才能跳过其他 user_id, 只遍历这一个 user_id 对应的记录
而悲催的是, 如果使用 :ets.fun2ms 函数来做的话, 生成的是这样的 match specification

1
2
iex> :ets.fun2ms(fn {{user_id, _}, pid} when user_id == 1 -> pid end)
[{{{:"$1", :_}, :"$2"}, [{:==, :"$1", 1}], [:"$2"]}]

这种写法将不会用到 ordered_set 的前缀匹配, 而是全表扫描, 是全表扫描!

两者性能差异视表的规模而定, 这也是坑点之一,

因为表小的时候看不出来, 等表大了以后又很难想到是这里的问题, 莫名奇妙 CPU 占用率就飙上去了

下面是一个大约 20 万元素的表的实际数据:

1
2
3
4
5
6
7
8
9
10
> :timer.tc(fn -> :ets.select(:simple_global, [ {{{:"$1", :agent, :"$2"}, :_, :_, :_, :_}, [{:==, :"$1", ent_id}], [:"$2"]} ]) end)
{12128,
 [9269, 44558, 1001458, 1068440, 1069147, 1092569, 1111688, 1114902, 1118485,
  1155156, 1161491, 1172293, 1183136, 1186487, 1228076, 1234522, 1268137,
  1273945, 1359669, 1488611, 1506535, 1513229, 1544955, 1555843, 1560291]}
> :timer.tc(fn -> :ets.select(:simple_global, [{{{ent_id, :agent, :"$2"}, :_, :_, :_, :_}, [], [:"$2"]}]) end)
{25,
 [9269, 44558, 1001458, 1068440, 1069147, 1092569, 1111688, 1114902, 1118485,
  1155156, 1161491, 1172293, 1183136, 1186487, 1228076, 1234522, 1268137,
  1273945, 1359669, 1488611, 1506535, 1513229, 1544955, 1555843, 1560291]}

12128 微秒和 25 微秒的差距, 约 500 倍,

笔者的某服务经此优化, CPU 占用率从 80% 降到了 20% !

展开全文 >>

erlang inet&tcp

gen_tcp:shutdown

函数配合{exit_on_close, false}选项可以实现tcp半开的效果

inet_db:lookup_socket

可以获取socket对应的模块

{ok, S} = gen_tcp:listen(10000, []).
{ok,#Port<0.4423655>}
inet_db:lookup_socket(S).
{ok,inet_tcp}
{ok, S2} = gen_udp:open(10001).
{ok,#Port<0.4429431>}
inet_db:lookup_socket(S2).
{ok,inet_udp}

inet_db:register_socket

下面这种做法没有直接调用gen_tcp:accept而是伪装成使用了gen_tcp:accept

gen_tcp:accept里面也就只是调了lookup_socket和register_socket
% patch up the socket so it looks like one we got from
% gen_tcp:accept/1
{ok, Mod} = inet_db:lookup_socket(LSock),
% 内部调用erlang:port_set_data记录信息
inet_db:register_socket(Sock, Mod),

inet:getaddr

检查是否支持ipv6

inet:getaddr("localhost", inet).
{ok,{127,0,0,1}}
inet:getaddr("localhost", inet6).
{error,nxdomain}

inet_parse:address

inet_parse:address("192.168.1.1").
{ok,{192,168,1,1}}

inet:sockname

inet:peername

inet_parse:ntoa

inet_parse:ntoa({192,168,1,113}).
"192.168.1.113"

inet:getstat

查socket的状态

inet:getif

1
2
3
4
5
{ok,[{{172,17,42,1},{0,0,0,0},{255,255,0,0}},
    {{172,16,205,1},{172,16,205,255},{255,255,255,0}},
    {{192,168,117,1},{192,168,117,255},{255,255,255,0}},
    {{192,168,1,114},{192,168,1,255},{255,255,255,0}},
    {{127,0,0,1},{0,0,0,0},{255,0,0,0}}]}

如何检查当前环境下的 sndbuf, recbuf, buffer 这些参数的默认值

OSX sysctl

1
2
3
4
5
6
7
8
9
10
sysctl -a | grep space
net.local.stream.sendspace: 8192
net.local.stream.recvspace: 8192
net.local.dgram.recvspace: 4096
net.inet.tcp.sendspace: 131072
net.inet.tcp.recvspace: 131072
net.inet.udp.recvspace: 786896
net.inet.raw.recvspace: 8192
net.stats.sendspace: 2048
net.stats.recvspace: 8192

OSX default

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Eshell V10.7  (abort with ^G)
1> {ok, LSock} = gen_tcp:listen(12345, []).
{ok,#Port<0.5>}
2> spawn(fun() -> {ok, Sock} = gen_tcp:accept(LSock), io:format("~p", [inet:getopts(Sock, [recbuf, sndbuf, buffer])]), receive _ -> ok end end).
<0.86.0>
3> inet:getopts(LSock, [recbuf, sndbuf, buffer]).
{ok,[{recbuf,131072},{sndbuf,131072},{buffer,1460}]}
4> {ok, S} = gen_tcp:connect("127.0.0.1", 12345, []).
{ok,#Port<0.7>}
{ok,[{recbuf,408300},{sndbuf,146988},{buffer,1460}]}
5> inet:getopts(S, [recbuf, sndbuf, buffer]).
{ok,[{recbuf,408300},{sndbuf,146988},{buffer,1460}]}

OSX 4096

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Eshell V10.7  (abort with ^G)
1> {ok, LSock} = gen_tcp:listen(12345, [{sndbuf, 4096}, {recbuf, 4096}]).
{ok,#Port<0.5>}
2> spawn(fun() -> {ok, Sock} = gen_tcp:accept(LSock), io:format("~p", [inet:getopts(Sock, [recbuf, sndbuf, buffer])]), receive _ -> ok end end).
<0.86.0>
3> inet:getopts(LSock, [recbuf, sndbuf, buffer]).
{ok,[{recbuf,4096},{sndbuf,4096},{buffer,4096}]}
4> {ok, S} = gen_tcp:connect("127.0.0.1", 12345, [{sndbuf, 4096}, {recbuf, 4096}]).
{ok,#Port<0.7>}
{ok,[{recbuf,326640},{sndbuf,65328},{buffer,4096}]}
5> inet:getopts(S, [recbuf, sndbuf, buffer]).
{ok,[{recbuf,326640},{sndbuf,65328},{buffer,4096}]}

Ubuntu

1
2
3
4
5
6
7
8
# cat /proc/sys/net/ipv4/tcp_rmem
4096    131072  6291456
# cat /proc/sys/net/ipv4/tcp_wmem
4096    16384   4194304
# cat /proc/sys/net/core/rmem_default
212992
# cat /proc/sys/net/core/wmem_default
212992

Ubuntu default

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Eshell V10.7  (abort with ^G)
1> {ok, LSock} = gen_tcp:listen(12345, []).
{ok,#Port<0.6>}
2> spawn(fun() -> {ok, Sock} = gen_tcp:accept(LSock), io:format("~p", [inet:getopts(Sock, [recbuf, sndbuf, buffer])]), receive _ -> okend end).
<0.83.0>
3> inet:getopts(LSock, [recbuf, sndbuf, buffer]).
{ok,[{recbuf,131072},{sndbuf,16384},{buffer,1460}]}
4> {ok, S} = gen_tcp:connect("127.0.0.1", 12345, []).
{ok,#Port<0.8>}
{ok,[{recbuf,131072},{sndbuf,2626560},{buffer,1460}]}
5> inet:getopts(S, [recbuf, sndbuf, buffer]).
{ok,[{recbuf,131072},{sndbuf,2626560},{buffer,1460}]}

Ubuntu 4096

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Erlang/OTP 22 [erts-10.7] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

Eshell V10.7  (abort with ^G)
1> {ok, LSock} = gen_tcp:listen(12345, [{recbuf,4096},{sndbuf,4096}]).
{ok,#Port<0.6>}
2> spawn(fun() -> {ok, Sock} = gen_tcp:accept(LSock), io:format("~p", [inet:getopts(Sock, [recbuf, sndbuf, buffer])]), receive _ -> okend end).
<0.83.0>
3> inet:getopts(LSock, [recbuf, sndbuf, buffer]).
{ok,[{recbuf,8192},{sndbuf,8192},{buffer,4096}]}
4> {ok, S} = gen_tcp:connect("127.0.0.1", 12345, [{recbuf,4096},{sndbuf,4096}]).
{ok,#Port<0.8>}
{ok,[{recbuf,8192},{sndbuf,8192},{buffer,4096}]}
5> inet:getopts(S, [recbuf, sndbuf, buffer]).
{ok,[{recbuf,8192},{sndbuf,8192},{buffer,4096}]}

展开全文 >>

ruby问题集

据说是由于ruby官网用的 ssl 从 sha1 改到 sha256 导致的问题(完整的背景见参考链接)

安装 gem 时出现如下问题

1
SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

命令行更新 ruby 也更新不了, 只能从官网下载了个 2.5.2 的 ruby 的安装包,
又装了个 rbenv (为了不和 macos 系统自带的 ruby 相冲突), 安装了 ruby 之后
才成功地 gem install 了几个 gem

more >>

展开全文 >>

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true