Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to content

增加新建column family操作-demo01 #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 19 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 20 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,65 +17,65 @@ This extension is developed and maintained by the VidarDB team. Feel free to rep
We test this foreign data wrapper on Ubuntu Server 18.04 using PostgreSQL-11 together with RocksDB-6.2.4 (built with GCC-7.4.0).

- Install PostgreSQL and the dev library which is required by extensions:

```sh
# add the repository
sudo tee /etc/apt/sources.list.d/pgdg.list << END
deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main
END

# get the signing key and import it
wget https://www.postgresql.org/media/keys/ACCC4CF8.asc
sudo apt-key add ACCC4CF8.asc

# fetch the metadata from the new repo
sudo apt-get update

# install postgresql and the dev library
sudo apt-get install postgresql-11
sudo apt-get install postgresql-server-dev-11
```

- Install [RocksDB](https://github.com/facebook/rocksdb) from source code:

```sh
git clone -b v6.2.4 https://github.com/facebook/rocksdb.git

cd rocksdb

sudo DEBUG_LEVEL=0 make shared_lib install-shared

sudo sh -c "echo /usr/local/lib >> /etc/ld.so.conf"

sudo ldconfig
```

- Build this foreign data wrapper:

```sh
git clone git@github.com:postgrespro/lsm.git

cd lsm

make

sudo make install
```

- Before using this foreign data wrapper, we need to add it to `shared_preload_libraries` in the `postgresql.conf`:

```sh
echo "shared_preload_libraries = 'lsm'" >> postgresql.conf
```

and restart PostgreSQL:

```sh
sudo service postgresql restart
```

- When uninstall this extension, first issue the following commands, and then delete the data by locating PostgreSQL data folder via `show data_directory;` in PostgreSQL terminal.

```sh
cd PostgresForeignDataWrapper

Expand All @@ -98,7 +98,6 @@ This extension does not have any parameter. After creating the extension and cor

A simple example is as follows (*you can run '`sudo -u postgres psql -U postgres`' to connect the local postgresql server*):


```
CREATE DATABASE example;
\c example
Expand All @@ -124,17 +123,15 @@ A simple example is as follows (*you can run '`sudo -u postgres psql -U postgres

DROP SERVER lsm_server;
DROP EXTENSION lsm_fdw;

\c postgres
DROP DATABASE example;

```
```

# Testing

We have tested certain typical SQL statements and will add more test cases later. The test scripts are in the test/sql folder which are recommended to be placed in a non-root directory. The corresponding results can be found in the test/expected folder. You can run the tests in the following way:


```sh
sudo service postgresql restart

Expand All @@ -147,16 +144,15 @@ We have tested certain typical SQL statements and will add more test cases later
sudo -u postgres psql -U postgres -d lsmtest -a -f test/sql/clear.sql
```

# Debug
# Debug

If you want to debug the source code, you may need to start PostgreSQL in the debug mode:


```sh
sudo service postgresql stop

sudo -u postgres /usr/lib/postgresql/11/bin/postgres -d 0 -D /var/lib/postgresql/11/main -c config_file=/etc/postgresql/11/main/postgresql.conf
```
```

# Docker

Expand Down
8 changes: 0 additions & 8 deletions lsm--0.1.sql

This file was deleted.

1 change: 0 additions & 1 deletion lsm.conf

This file was deleted.

5 changes: 0 additions & 5 deletions lsm.control

This file was deleted.

12 changes: 7 additions & 5 deletions lsm_api.h
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,27 @@
* This file represents API between client working with Postgres API and C++ server working with RocksDB API.
* To avoid collision between C/C++ headers they are sharing just this one header file.
*/
#ifndef __LSM_API_H__
#define __LSM_API_H__
#ifndef __LSM_API_H__ // 如果没有定义这个宏
#define __LSM_API_H__ // 就定义下面的宏

#include <stddef.h>
#include <stdint.h>
#include <stdbool.h>

#ifdef __cplusplus
extern "C" {
extern "C" { //加上extern "C"后,会指示编译器这部分代码按C语言的进行编译,而不是C++的。
#endif

/*
* Maximal size of record which can be transfered through client-server protocol and read batch size as well
*/
#define LSM_MAX_RECORD_SIZE (64*1024)
// 定义client 和 server之间传送的数据的最大数据量
#define LSM_MAX_RECORD_SIZE (64*1024) //64KB

/*
* Name of the directory in $PGDATA
*/
#define LSM_FDW_NAME "lsm"
#define LSM_FDW_NAME "lsm" //定义fdw的名称为lsm

extern int LsmQueueSize;
extern bool LsmSync;
Expand All @@ -42,6 +43,7 @@ typedef enum
LsmOpLookup
} LsmOperation;

// 定义client和server之间的各种操作
extern void LsmError(char const* message);
extern size_t LsmShmemSize(int maxClients);
extern void LsmInitialize(void* ctl, int maxClients);
Expand Down
10 changes: 7 additions & 3 deletions lsm_client.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -57,15 +57,17 @@ LsmDelete(LsmQueueId qid, LsmRelationId rid, char *key, size_t keyLen)
return true;
}


// 将传入的数据以LsmMessage进行传输
bool
LsmInsert(LsmQueueId qid, LsmRelationId rid, char *key, size_t keyLen, char *val, size_t valLen)
{
LsmMessage msg;
LsmQueue* queue = queues[qid];
msg.hdr.op = LsmOpInsert;
msg.hdr.rid = rid;
msg.hdr.keySize = keyLen;
msg.hdr.valueSize = valLen;
msg.hdr.rid = rid; //外部表oid
msg.hdr.keySize = keyLen; //key的长度
msg.hdr.valueSize = valLen; //value的长度
msg.key = key;
msg.value = val;
queue->put(msg);
Expand All @@ -77,6 +79,8 @@ LsmInsert(LsmQueueId qid, LsmRelationId rid, char *key, size_t keyLen, char *val
return true;
}



uint64_t
LsmCount(LsmQueueId qid, LsmRelationId rid)
{
Expand Down
18 changes: 15 additions & 3 deletions lsm_db.h
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,13 @@

using namespace rocksdb;

/**
* 对Rocksdb的封装
*
* /

/*
* Wrapper for RocksSB
* Wrapper for RocksDB
*/
struct LsmConnection
{
Expand Down Expand Up @@ -44,6 +49,7 @@ struct LsmMessageHeader
/*
* Protocol message
*/
// 用于包装传输的数据
struct LsmMessage
{
LsmMessageHeader hdr;
Expand All @@ -54,8 +60,10 @@ struct LsmMessage
/*
* Queue for tranferring data between backend and LSM worker thread.
*/
// 用于将数据传送到rocksdb中
struct LsmQueue
{
// ring buffer 环状缓冲区
volatile int getPos; // get position in ring buffer (updated only by consumer)
volatile int putPos; // put position in ring buffer (updated only by producer)
volatile int respSize; // response size
Expand Down Expand Up @@ -84,11 +92,13 @@ struct LsmCursor

struct LsmServer;


// 主要封装了对lsm的操作,也就是lsmServer中的一个工作进程
struct LsmWorker
{
std::map<LsmCursorId, LsmCursor> cursors;
LsmServer* server;
LsmQueue* queue;
LsmServer* server; // 一个server有很多的worker
LsmQueue* queue; //一个worker对应一个queue
pthread_t thread;

LsmWorker(LsmServer* s, LsmQueue* q) : server(s), queue(q) {}
Expand Down Expand Up @@ -150,6 +160,8 @@ class CriticalSection
}
};


// 此结构体包含了很多对rocksdb的操作,非常重要,是一个对于rocksdb封装的最大对象,包含了很多的LsmWorker
struct LsmServer
{
LsmWorker** workers;
Expand Down
Loading