smlar

Author: digoal

201502/20150228_01.md

@@ -0,0 +1,3 @@
+### 文章列表  
+----  
+##### 20150228_01.md   [《PostgreSQL 统计信息之 - 逻辑与物理存储的线性相关性》](20150228_01.md)  

201502/20150228_01_pic_001.png

@@ -0,0 +1,98 @@
+## PostgreSQL 计算 任意类型 字段之间的线性相关性  
+                                                                                                                      
+### 作者                                                                                                                          
+digoal                                                                                                                          
+                                                                                                                      
+### 日期                                                                                                                          
+2016-04-03                                                                                                                     
+                                                                                                                      
+### 标签                                                                                                                          
+PostgreSQL , 线性相关性                                                
+                                                                                                                      
+----                                                                                                                          
+                               
+## 背景      
+PostgreSQL自带了计算numeric和numeric字段的线性相关性的聚合函数corr(numeric, numeric)。    
+  
+例如：    
+  
+```  
+postgres=# select corr(c1,c2) from (values (1,2),(2,1),(100,90),(13,13),(25,27) ) t(c1,c2);  
+       corr          
+-------------------  
+ 0.998528203831946  
+(1 row)  
+  
+postgres=# \df+ corr  
+                                                                             List of functions  
+   Schema   | Name | Result data type |        Argument data types         | Type | Security | Volatility |  Owner   | Language |   Source code   |       Description         
+------------+------+------------------+------------------------------------+------+----------+------------+----------+----------+-----------------+-------------------------  
+ pg_catalog | corr | double precision | double precision, double precision | agg  | invoker  | immutable  | postgres | internal | aggregate_dummy | correlation coefficient  
+(1 row)  
+```  
+  
+如果要计算多元的线性相关性，可以使用madlib提供的linregr_train函数来统计。    
+  
+http://doc.madlib.net/latest/group__grp__linreg.html    
+  
+注意不管是一元回归还是多元回归，都需要提供数字类型，如果是文本是不支持的，如下：    
+  
+```  
+postgres=# select corr(c1,c3) from (values (1,2,'test'),(2,1,'digoal'),(100,90,'hello'),(13,13,'china'),(25,27,'hangzhou') ) t(c1,c2,c3);  
+ERROR:  function corr(integer, text) does not exist  
+LINE 1: select corr(c1,c3) from (values (1,2,'test'),(2,1,'digoal'),...  
+               ^  
+HINT:  No function matches the given name and argument types. You might need to add explicit type casts.  
+```  
+  
+那么怎么处理呢？    
+  
+PostgreSQL提供了强大的窗口功能，因为任意字段都可以排序，所以只要使用窗口输出字段排序后的rank()就可以代表它的位置从而计算相关性。    
+  
+例如    
+  
+```  
+postgres=# select   
+c1, rank() over(order by c1) rc1,   
+c2, rank() over(order by c2) rc2,   
+c3, rank() over(order by c3) rc3   
+  from (values (1,2,'test'),(2,1,'digoal'),(100,90,'hello'),(13,13,'china'),(25,27,'hangzhou') )   
+t(c1,c2,c3) order by c1;   
+ c1  | rc1 | c2 | rc2 |    c3    | rc3   
+-----+-----+----+-----+----------+-----  
+   1 |   1 |  2 |   2 | test     |   5  
+   2 |   2 |  1 |   1 | digoal   |   2  
+  13 |   3 | 13 |   3 | china    |   1  
+  25 |   4 | 27 |   4 | hangzhou |   3  
+ 100 |   5 | 90 |   5 | hello    |   4  
+(5 rows)  
+```  
+  
+这个例子要计算c1,c3的相关性，c1是数字字段，但是c3是text。 corr函数不支持这么操作。    
+  
+因此我使用上面这条带窗口的SQL，把text字段根据rank抽象为数值，正好和其他字段可以匹配相关性。  
+  
+来看计算结果：    
+  
+```  
+postgres=# select corr(c1,rc3), corr(rc1,rc3) from (  
+  select   
+    c1, rank() over(order by c1) rc1,   
+    c2, rank() over(order by c2) rc2,   
+    c3, rank() over(order by c3) rc3   
+  from (values (1,2,'test'),(2,1,'digoal'),(100,90,'hello'),(13,13,'china'),(25,27,'hangzhou')   
+  ) t(c1,c2,c3)   
+) t;  
+       corr        | corr   
+-------------------+------  
+ 0.283302495025433 | -0.1  
+(1 row)  
+```  
+  
+建议采用corr(rc1,rc3)的结果，这个比较有代表性。 代表被评测列的线性相关性。     
+  
+应用场景大家猜一猜。    
+                                                      
+[Count](http://info.flagcounter.com/h9V1)                                                                                                                          
+                                                                        
+                                                                    

201502/readme.md

@@ -1,6 +1,7 @@
 ### 文章列表  
 ----  
 ##### 20160401_01.md   [《阿里云 PostgreSQL pg_hint_plan插件的用法》](20160401_01.md)  
+##### 20160403_01.md   [《PostgreSQL 计算 任意类型 字段之间的线性相关性》](20160403_01.md)  
 ##### 20160414_01.md   [《PostgreSQL 物联网黑科技 - 瘦身几百倍的索引(BRIN index)》](20160414_01.md)  
 ##### 20160419_01.md   [《PostgreSQL 行级 全文检索》](20160419_01.md)  
 ##### 20160426_01.md   [《iperf 测试网络性能指标》](20160426_01.md)  

201604/20160403_01.md

@@ -0,0 +1,222 @@
+## 数据库小程序 - 广播(notify/listen)  
+                     
+### 作者                 
+digoal     
+      
+### 日期                                                                                                       
+2017-01-16                                                                                                             
+            
+### 标签                 
+PostgreSQL , notify , listen , 异步消息           
+      
+----     
+        
+## 背景    
+小时候就梦想有个酷酷的电波表(虽然现在还没有拥有)，不过电波表和PostgreSQL有什么关系呢？听我道来。  
+  
+![pic](20170116_01_pic_001.jpg)   
+  
+http://baike.baidu.com/view/1124741.htm  
+  
+电波表内置高感度小型天线，接收标准电波进行自动对时，因而可以实现时间上的精准。在国际上，德国、英国、美国、日本都已经有标准电波的发送。2007年7月，在中国河南商丘建成的新电波塔已经开始发送电波。  
+  
+标准电波接收（6局电波接收）原理   
+  
+中国标准时间以10万年误差1秒的铯原子钟为基准。通过手表内置的高敏感度接收器接收以无线电波传送的标准时间信号，并自动校准手表走时。使手表显示的时间与标准时间同步，精确计时。在建筑物密集的室内也可以接收标准电波信号，显示正确的时间。  
+  
+标准电波的接收范围是半径约1000~3000km, 深夜（最多6次）自动接收电波信号,不能收到标准电波信号时，自动接收GPS卫星电波获取时间信息并校正时间。中国的电波塔在河南省商丘市。  
+  
+而用于接收电波信号的防震天线采用的是非晶材质，能够实现高敏感度接收。不但具备对应极端户外环境下使用的牢靠性，还可以确保稳定接收全球6局电波信号。  
+  
+电波表是一个非常典型的广播应用，类似的还有组播（注意不是主播哦），类似的应用也很多，比如广播电视，电台等。  
+  
+## 数据库广播  
+在数据库中，其实也有类似的应用，比如我前几天在文章中写的数据库到WEB客户端的广播，详见 :   
+  
+[《从微信小程序 到 数据库"小程序" , 鬼知道我经历了什么》](./20170113_03.md)  
+  
+实际上是利用了PostgreSQL数据库的异步消息机制，数据库往消息通道发送数据，应用程序可以监听对应的消息通道，获取数据库发出的数据。  
+  
+通过异步消息在数据库中实现了一对多的广播效果。  
+  
+SQL语法参考文档：  
+  
+1\. 向通道发送消息  
+  
+https://www.postgresql.org/docs/9.6/static/sql-notify.html  
+  
+2\. 监听某通道的消息  
+  
+https://www.postgresql.org/docs/9.6/static/sql-listen.html  
+  
+3\. 取消监听某通道  
+  
+https://www.postgresql.org/docs/9.6/static/sql-unlisten.html  
+  
+4\. 数据库函数  
+  
+查看会话监听了哪些通道，以及当前数据库的异步消息队列使用了多少。  
+  
+```  
+pg_listening_channels()	        setof text	channel names that the session is currently listening on  
+pg_notification_queue_usage()	double	        fraction of the asynchronous notification queue currently occupied (0-1)  
+```  
+  
+pg_notification_queue_usage用来计算已使用的异步消息页面占比，如果有监听，但是一直不消费，可能导致溢出。  
+  
+```  
+src/backend/commands/async.c  
+  
+/*  
+ * slru.c currently assumes that all filenames are four characters of hex  
+ * digits. That means that we can use segments 0000 through FFFF.  
+ * Each segment contains SLRU_PAGES_PER_SEGMENT pages which gives us  
+ * the pages from 0 to SLRU_PAGES_PER_SEGMENT * 0x10000 - 1.  
+ *  
+ * It's of course possible to enhance slru.c, but this gives us so much  
+ * space already that it doesn't seem worth the trouble.  
+ *  
+ * The most data we can have in the queue at a time is QUEUE_MAX_PAGE/2  
+ * pages, because more than that would confuse slru.c into thinking there  
+ * was a wraparound condition.  With the default BLCKSZ this means there  
+ * can be up to 8GB of queued-and-not-read data.  
+ *  
+ * Note: it's possible to redefine QUEUE_MAX_PAGE with a smaller multiple of  
+ * SLRU_PAGES_PER_SEGMENT, for easier testing of queue-full behaviour.  
+ */  
+#define QUEUE_MAX_PAGE                  (SLRU_PAGES_PER_SEGMENT * 0x10000 - 1)  
+  
+  
+src/include/access/slru.h:#define SLRU_PAGES_PER_SEGMENT        32  
+```  
+  
+## 异步消息编程  
+除了使用SQL来编写异步消息，还可以使用数据库的驱动来编写异步消息  
+  
+### c  
+参考libpq的异步消息部分  
+  
+https://www.postgresql.org/docs/9.6/static/libpq-notify.html  
+  
+```  
+PGnotify *PQnotifies(PGconn *conn);  
+  
+typedef struct pgNotify  
+{  
+    char *relname;              /* notification channel name */  
+    int  be_pid;                /* process ID of notifying server process */  
+    char *extra;                /* notification payload string */  
+} PGnotify;  
+```  
+  
+文档中有一个例子如下  
+  
+https://www.postgresql.org/docs/9.6/static/libpq-example.html#LIBPQ-EXAMPLE-2  
+  
+  
+### java  
+参考文档  
+  
+https://jdbc.postgresql.org/documentation/head/listennotify.html  
+  
+### 应用举例  
+Broadcasting PostgreSQL NOTIFY messages to WebSocket Clients    
+  
+The system works like this:  
+  
+```
+Client subscribes to a WebSocket topic...  
+  
+NOTIFY event on database server ->  
+  PGNotificationListener on web server ->  
+      Send Websocket notification on server ->  
+         Receive Websocket event on browser.   
+```
+  
+With the code below, if you call NOTIFY dml_events, 'some message'; in Postgres, it will be broadcast to all WebSocket clients.  
+  
+Follow this [answer](https://bitbucket.org/neilmcg/postgresql-websocket-example)  regarding proper listener setup  
+  
+URL：   
+  
+http://blog.databasepatterns.com/2014/04/postgresql-nofify-websocket-spring-mvc.html  
+  
+http://stackoverflow.com/questions/37916489/listen-notify-pgconnection-goes-down-java  
+  
+```  
+The notification listeners are internally maintained by that library as weak references meaning that you have to hold a hard reference externally so they won't be garbage collected.   
+Check out the BasicContext class lines 642 - 655:  
+  
+---  
+public void addNotificationListener(String name, String channelNameFilter, NotificationListener listener) {  
+  
+    name = nullToEmpty(name);  
+    channelNameFilter = channelNameFilter != null ? channelNameFilter : ".*";  
+  
+    Pattern channelNameFilterPattern = Pattern.compile(channelNameFilter);  
+  
+    NotificationKey key = new NotificationKey(name, channelNameFilterPattern);  
+  
+    synchronized (notificationListeners) {  
+      notificationListeners.put(key, new WeakReference<NotificationListener>(listener));  
+    }  
+  
+}  
+---  
+  
+If the GC picks up your listener, calls to "get" on the weak reference will return null and will not fire as seen from lines 690 - 710  
+  
+---  
+  @Override  
+  public synchronized void reportNotification(int processId, String channelName, String payload) {  
+  
+    Iterator<Map.Entry<NotificationKey, WeakReference<NotificationListener>>> iter = notificationListeners.entrySet().iterator();  
+    while (iter.hasNext()) {  
+  
+      Map.Entry<NotificationKey, WeakReference<NotificationListener>> entry = iter.next();  
+  
+      NotificationListener listener = entry.getValue().get();  
+      if (listener == null) {  
+  
+        iter.remove();  
+      }  
+      else if (entry.getKey().channelNameFilter.matcher(channelName).matches()) {  
+  
+        listener.notification(processId, channelName, payload);  
+      }  
+  
+    }  
+  
+}  
+---  
+  
+To fix this, add your notification listeners as such:  
+  
+---  
+/// Do not let this reference go out of scope!  
+PGNotificationListener listener = new PGNotificationListener() {  
+  
+@Override  
+public void notification(int processId, String channelName, String payload) {  
+    // interesting code  
+};  
+pgConnection.addNotificationListener(listener);  
+---  
+  
+Quite an odd use-case for weak references in my opinion...  
+```  
+  
+代码：  
+  
+https://bitbucket.org/neilmcg/postgresql-websocket-example  
+    
+其他编程语言的驱动，大多数是基于libpq的，不再举例。  
+  
+## 参考  
+  
+[《从微信小程序 到 数据库"小程序" , 鬼知道我经历了什么》](./20170113_03.md)  
+  
+  
+        
+[Count](http://info.flagcounter.com/h9V1)                                                                                                    
+