issue=1258, t-cache support block-level cache evict #1266

caijieming-ng · 2017-05-18T18:26:00Z

No description provided.

baidubot · 2017-05-18T18:26:18Z

caijieming-ng · 2017-06-12T08:36:52Z

build

yvxiang · 2017-07-25T03:26:23Z

src/leveldb/util/block_cache.cc

+        for (uint32_t i = 0; i < c_valid.size(); ++i) {
+            MutexLock lockgard(&cache_->mu_);
+            CacheBlock* block = c_valid[i];
+            block->cv.Wait();


这里也得是while吧

yvxiang · 2017-08-08T02:24:44Z

src/io/utils_leveldb.cc

@@ -18,6 +18,7 @@
 #include "leveldb/comparator.h"
 #include "leveldb/env_dfs.h"
 #include "leveldb/env_flash.h"
+#include "leveldb/block_cache.h"


目测顺序有问题

yvxiang · 2017-08-08T02:25:00Z

src/io/utils_leveldb.cc

@@ -31,6 +32,7 @@ DECLARE_string(tera_leveldb_env_hdfs2_nameservice_list);
 DECLARE_string(tera_tabletnode_path_prefix);
 DECLARE_string(tera_dfs_so_path);
 DECLARE_string(tera_dfs_conf);
+DECLARE_int32(tera_leveldb_block_cache_env_num_thread);


thread_num?

yvxiang · 2017-08-08T02:29:20Z

src/io/utils_leveldb.cc

@@ -66,6 +68,21 @@ leveldb::Env* LeveldbBaseEnv() {
    }
 }

+// Tcache: default env
+static pthread_once_t block_cache_once = PTHREAD_ONCE_INIT;
+static leveldb::Env* default_block_cache_env;


为啥不像mem及flash一样，弄成函数内的静态变量

是静态的

yvxiang · 2017-08-08T02:32:32Z

src/leveldb/include/leveldb/block_cache.h

+// found in the LICENSE file.
+
+#ifndef  STOREAGE_LEVELDB_UTIL_BLOCK_CACHE_H
+#define  STOREAGE_LEVELDB_UTIL_BLOCK_CACHE_H


STOREAGE_LEVELDB_UTIL_BLOCK_CACHE_H_？

yvxiang · 2017-08-08T02:43:20Z

src/tabletnode/tabletnode_impl.cc

+    // compitable with legacy FlashEnv
+    leveldb::FlashEnv* flash_env = (leveldb::FlashEnv*)io::LeveldbFlashEnv();
+    flash_env->SetFlashPath(FLAGS_tera_tabletnode_cache_paths,
+            FLAGS_tera_io_cache_path_vanish_allowed);


对齐好像有问题？

yvxiang · 2017-08-08T03:04:31Z

src/leveldb/util/cache.cc

+    return reinterpret_cast<LRUHandle*>(handle)->value;
+  }
+
+  uint64_t NewId() {


这个是用来干嘛的？

这个接口在block cache中没有用

yvxiang · 2017-08-08T03:14:08Z

src/leveldb/include/leveldb/cache.h

@@ -29,9 +29,36 @@ namespace leveldb {

 class Cache;

+// An entry is a variable length heap-allocated structure.  Entries
+// are kept in a circular doubly linked list ordered by access time.
+struct LRUHandle {


挪出来的目的是？

能填充cache_id

yvxiang · 2017-08-08T03:15:24Z

src/leveldb/include/leveldb/block_cache.h

+Env* NewBlockCacheEnv(Env* base);
+
+} // leveldb
+#endif


STOREAGE_LEVELDB_UTIL_BLOCK_CACHE_H_

yvxiang · 2017-08-08T03:16:18Z

src/leveldb/util/cache.cc

+  // Dummy head of LRU list.
+  // lru.prev is newest entry, lru.next is oldest entry.
+  //LRUHandle hot_lru_;
+  //LRUHandle cold_lru_;


咋感觉像内核里的page cache：）

yvxiang · 2017-08-08T03:26:43Z

src/leveldb/util/cache.cc

@@ -286,6 +261,170 @@ size_t LRUCache::TotalCharge() {
  return usage_;
 }

+class LRU2QCache: public Cache {


没太明白和原生LRUCache的区别，除了在找坑方面有区别，其它的有吗？

原生lru是可超了，这个lru是描述持久化的ssd上block的lru，不能超

yvxiang · 2017-08-08T06:06:43Z

src/leveldb/util/block_cache.cc

+// Tcache
+/////////////////////////////////////////////
+uint64_t kBlockSize = 4096UL;
+uint64_t kDataSetSize = 134217728UL;


搞成128 * 1024 * 1024或者128 << 20这种？

yvxiang · 2017-08-08T06:25:35Z

src/leveldb/util/block_cache.cc

+
+    Status NewRandomAccessFile(const std::string& fname,
+                               RandomAccessFile** result); // cache Pread
+    static void BlockDeleter(const Slice& key, void* v);


和前面比少一个空行？

yvxiang · 2017-08-08T06:40:20Z

src/leveldb/util/block_cache.cc

+    BlockCacheImpl* cache = new BlockCacheImpl(options);
+    Status s = cache->LoadCache();
+    assert(s.ok());
+    cache_vec_.push_back(cache); // no need lock


前面调Load时没打算搞成多线程的？

这是ts启动时，加载cache（没有开销），不是load tablet的加载；

yvxiang · 2017-08-08T06:50:03Z

src/leveldb/util/block_cache.cc

+        MutexLock l(&mu_);
+        if (tmp_storage_ == NULL) {
+            tmp_storage_ = new std::string();
+            tmp_storage_->resize(0);


这货自己就是0吧

yvxiang · 2017-08-08T07:01:34Z

src/leveldb/util/block_cache.cc

+            tmp_storage_->resize(0);
+            block_list_.push_back(tmp_storage_);
+        }
+        uint32_t begin = offset_ / block_size_;


有没有必要把后面抠到锁外

yvxiang · 2017-08-08T07:06:21Z

src/leveldb/util/block_cache.cc

+            Slice buf(data.data() + tmp_size, data.size() - tmp_size);
+            for (uint32_t i = begin + 1; i <= end; ++i) {
+                tmp_storage_ = new std::string();
+                tmp_storage_->resize(0);


yvxiang · 2017-08-08T07:11:02Z

src/leveldb/util/block_cache.cc

+                tmp_storage_ = new std::string();
+                tmp_storage_->resize(0);
+                block_list_.push_back(tmp_storage_);
+                if (i < end) { // last block


注释应该是写错了

yvxiang · 2017-08-08T07:12:06Z

src/leveldb/util/block_cache.cc

+            for (uint32_t i = begin + 1; i <= end; ++i) {
+                tmp_storage_ = new std::string();
+                tmp_storage_->resize(0);
+                block_list_.push_back(tmp_storage_);


有没有必要放掉锁，一波流扔后台去刷

yvxiang · 2017-08-08T07:35:35Z

src/leveldb/util/block_cache.cc

+        Log("[%s] begin release %s\n", cache_->WorkPath().c_str(), fname_.c_str());
+        MutexLock lockgard(&cache_->mu_);
+        uint64_t block_idx;
+        std::string* block_data = write_buffer_.PopBackBlock(&block_idx);


为啥是拿后面的？

因为前面的块是通过BGFlush异步下刷的（按block size粒度，不满足这个粒度的，BGFlush不处理），只有close的时候，才能知道文件确实结束了，所有在此时刷最后一个不满粒度的数据块

yvxiang · 2017-08-08T07:36:53Z

src/leveldb/util/block_cache.cc

+            dfs_file_ = NULL;
+        }
+
+        Log("[%s] begin release %s\n", cache_->WorkPath().c_str(), fname_.c_str());


这块直接调一把close可以不，感觉重了

yvxiang · 2017-08-09T06:06:22Z

src/leveldb/util/block_cache.cc

+public:
+    BlockCacheWritableFile(BlockCacheImpl* c, const std::string& fname, Status* s)
+        : cache_(c),
+          bg_cv_(&c->mu_),


不暴露出来这个锁可以吗？由BlockCacheImpl提供一个阻塞的接口

yvxiang · 2017-08-09T06:06:34Z

src/leveldb/util/block_cache.cc

+            cache_->options_.block_size,
+            s->ToString().c_str());
+
+        MutexLock lockgard(&cache_->mu_);


同上，感觉有点怪

yvxiang · 2017-08-09T06:13:32Z

src/leveldb/util/block_cache.cc

+    mu_.AssertHeld();
+    uint64_t fid = 0;
+    std::string key = "FNAME#" + fname;
+    mu_.Unlock();


感觉略怪，为啥这里突然把锁放了，而调的时候前面特意加个锁

yvxiang · 2017-08-09T06:41:49Z

src/leveldb/util/block_cache.cc

+
+    Waiter* w = NULL;
+    LockKeyMap::iterator it = lock_key_.find(key);
+    if (it != lock_key_.end()){


目测少个空格

yvxiang · 2017-08-09T06:48:05Z

src/leveldb/util/block_cache.cc

+        }
+        write_buffer_.Append(data);
+
+        MutexLock lockgard(&cache_->mu_);


为啥这里要拿锁？

yvxiang · 2017-08-09T06:51:06Z

src/leveldb/util/block_cache.cc

+        Log("[%s] Begin BGFlush: %s\n", cache_->WorkPath().c_str(), fname_.c_str());
+        MutexLock lockgard(&cache_->mu_);
+        uint64_t block_idx;
+        std::string* block_data = write_buffer_.PopFrontBlock(&block_idx);


前面扔了好多task，这里不搞成while一波流主要是为了并发考虑？

yvxiang · 2017-08-09T06:57:43Z

src/leveldb/util/block_cache.cc

+            port::CondVar cv(&cache_->mu_);
+            cv.Wait(10); // timewait 10ms retry
+        }
+        block->state = 0;


把0也搞成个前面那砦枚举的状态？

可以改成断言，此时这个block必然是个无效块

yvxiang · 2017-08-09T08:34:55Z

src/leveldb/util/block_cache.cc

+            s->ToString().c_str());
+
+        MutexLock lockgard(&cache_->mu_);
+        fid_ = cache_->FileId(fname_);


如前，感觉锁略怪

yvxiang · 2017-08-09T08:44:38Z

src/leveldb/util/block_cache.cc

+                s = block->s; // degrade read
+            }
+            block->Clear(kCacheBlockLocked);
+            block->cv.SignalAll();


这里需要唤醒谁？

唤醒其他线程的c_lock中等待的block

yvxiang · 2017-08-09T08:45:20Z

src/leveldb/util/block_cache.cc

+
+            if (!block->Test(kCacheBlockLocked) &&
+                block->Test(kCacheBlockValid)) {
+                block->Set(kCacheBlockLocked | kCacheBlockCacheRead);


哈哈，感觉像内核玩法了：）

yvxiang · 2017-08-09T08:51:23Z

src/leveldb/util/block_cache.cc

+            block->WaitOnClear(kCacheBlockDfsRead);
+            block->Set(kCacheBlockCacheFill);
+            if (!block->s.ok() && s.ok()) {
+                s = block->s; // degrade read


如果这里失败，是不是后面可以直接跳到dfs->Read了

是的，走降级读模式

yvxiang · 2017-08-09T09:06:59Z

src/leveldb/util/block_cache.cc

+    new_fid_ = prev_fid_ + options_.fid_batch_num;
+    Log("[block_cache %s]: reuse block cache: prev_fid: %lu, new_fid: %lu\n",
+        dbname.c_str(), prev_fid_, new_fid_);
+    s = Status::OK();


强行OK？

yvxiang · 2017-08-09T09:18:17Z

src/leveldb/util/block_cache.cc

+uint64_t BlockCacheImpl::AllocFileId() { // no more than fid_batch_num
+    mu_.AssertHeld();
+    uint64_t fid = ++new_fid_;
+    while (new_fid_ - prev_fid_ >= options_.fid_batch_num) {


while在这里的作用是什么？

防止leveldb写失败

yvxiang · 2017-08-10T02:09:02Z

src/leveldb/util/block_cache.cc

+                PutFixed64(&hkey, block->block_idx);
+                block->sid = lc.sid;
+                block->cache_block_idx = DecodeFixed64(lkey.data());
+                block->state = (block->Test(kCacheBlockValid)) ? kCacheBlockValid : 0;


0换个方式表式？

yvxiang · 2017-08-10T02:35:45Z

src/leveldb/util/block_cache.cc

+                lc.KeyToString().c_str(),
+                lc.ValToString().c_str(),
+                s.ToString().c_str());
+        } else if (lc.type == kDataSetKey) {


这个分支实际上是一个load操作，是不是放别处比较好

这个函数主要维护一个lock table，给整个blockcacheimpl内部对象的并发控制使用；比如leveldb的读修改写，或并发的data set file的load，都走同一套并发控制逻辑；

bugfix: 1. cache reload core 2. support aio engine 3. cache fill TEST PASS

bluebore · 2017-08-22T03:16:36Z

src/leveldb/include/leveldb/block_cache.h

@@ -0,0 +1,101 @@
+// Copyright (c) 2015, Baidu.com, Inc. All Rights Reserved


这个看起来好别扭，既然是新文件，为啥会有个2015

bluebore · 2017-08-22T03:18:33Z

src/leveldb/include/leveldb/statistics.h

+namespace leveldb {
+
+/**
+ * Keep adding ticker's here.


这些代码是其他地方抄过来的么，质量有保证么，会不会带来版权问题？

哈哈被识破了感觉

删掉不需要的统计项，从rocksdb的性能统计方式参考的， :-(

这种方式调性能问题，比日志方式有效；glog和leveldb的打一条日志，大概在10微妙，很费。

bluebore · 2017-08-22T03:19:20Z

src/leveldb/util/cache.cc

@@ -25,31 +25,6 @@ namespace {

 // LRU cache implementation

-// An entry is a variable length heap-allocated structure.  Entries


这个为啥要删掉，挪走了？

1. lock optimize

taocp · 2017-09-07T11:20:34Z

您的执著和坚持，令人感动！继续加油~

fxsjy · 2017-09-07T13:37:55Z

同赞

bugfix for cache evict

baidubot assigned 00k May 18, 2017

caijieming-ng force-pushed the show_bug branch 3 times, most recently from 72a31b8 to 140eabf Compare May 18, 2017 18:51

yvxiang reviewed Jul 25, 2017

View reviewed changes

issue=1258, Tcache support block-level cache evict

177cb1e

caijieming-ng force-pushed the show_bug branch from 27af669 to 177cb1e Compare August 3, 2017 14:52

caijieming added 5 commits August 7, 2017 20:35

issue=1258, Tcache support block-level cache evict

3256bd2

issue=1258, Tcache support block-level cache evict

c5e3f83

issue=1258, Tcache support block-level cache evict

9b00baa

issue=1258, Tcache support block-level cache evict

170318f

issue=1258, Tcache support block-level cache evict

a1e51d3

yvxiang reviewed Aug 8, 2017

View reviewed changes

yvxiang reviewed Aug 9, 2017

View reviewed changes

yvxiang reviewed Aug 10, 2017

View reviewed changes

issue=1258, Tcache support block-level cache evict

b749896

bugfix: 1. cache reload core 2. support aio engine 3. cache fill TEST PASS

caijieming-ng changed the title ~~issue=1258, Tcache support block-level cache evict~~ issue=1258, t-cache support block-level cache evict Aug 21, 2017

caijieming added 3 commits August 22, 2017 01:41

issue=1258, Tcache support block-level cache evict

716270e

bugfix: 1. cache reload core 2. support aio engine 3. cache fill TEST PASS

issue=1258, Tcache support block-level cache evict

e516c24

issue=1258, Tcache support block-level cache evict

a94781f

bluebore reviewed Aug 22, 2017

View reviewed changes

issue=1258, Tcache support block-level cache evict

1c4682e

caijieming added 4 commits August 24, 2017 00:18

issue=1258, Tcache support block-level cache evict

749296e

issue=1258, Tcache support block-level cache evict

ae0298a

issue=1258, Tcache support block-level cache evict

415d026

issue=1258, Tcache support block-level cache evict

9794ff4

caijieming-ng assigned taocp Aug 29, 2017

caijieming added 2 commits August 30, 2017 15:29

issue=1258, t-cache support block-level cache evict

13813f3

1. lock optimize

issue=1258, t-cache support block-level cache evict

d95181f

1. lock optimize

caijieming added 2 commits October 24, 2017 23:50

issue=1258, t-cache support block-level cache evict

6394ef7

bugfix for cache evict

issue=1258, t-cache support block-level cache evict

9dc1665

bugfix for cache evict

		@@ -0,0 +1,101 @@
		// Copyright (c) 2015, Baidu.com, Inc. All Rights Reserved

		@@ -25,31 +25,6 @@ namespace {

		// LRU cache implementation

		// An entry is a variable length heap-allocated structure. Entries

issue=1258, t-cache support block-level cache evict #1266

Are you sure you want to change the base?

issue=1258, t-cache support block-level cache evict #1266

Conversation

caijieming-ng commented May 18, 2017

baidubot commented May 18, 2017

caijieming-ng commented Jun 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caijieming-ng Aug 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caijieming-ng Aug 21, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caijieming-ng Aug 22, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taocp commented Sep 7, 2017

fxsjy commented Sep 7, 2017

caijieming-ng Aug 21, 2017 •

edited

Loading

caijieming-ng Aug 21, 2017 •

edited

Loading

caijieming-ng Aug 22, 2017 •

edited

Loading