You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Multi-master is an extension and set of patches to a Postegres database, that turns Postgres into a
4
-
synchronous shared-nothing cluster to provide OLTP scalability and high availability with automatic
5
-
disaster recovery.
3
+
Multi-master is an extension and set of patches to a Postegres database, that turns Postgres into a synchronous shared-nothing cluster to provide OLTP scalability and high availability with automatic disaster recovery.
4
+
6
5
7
6
## Features
8
7
@@ -12,150 +11,54 @@ disaster recovery.
12
11
* Fault tolerance
13
12
* Automatic node recovery
14
13
15
-
## Overview
16
-
17
-
Multi-master replicates same database to all nodes in cluster and allows writes to each node. Transaction
18
-
isolation is enforced cluster-wide, so in case of concurrent updates on different nodes database will use the
19
-
same conflict resolution rules (mvcc with repeatable read isolation level) as single node uses for concurrent
20
-
backends and always stays in consistent state. Any writing transaction will write to all nodes, hence increasing
21
-
commit latency for amount of time proportional to roundtrip between nodes nedded for synchronization. Read only
22
-
transactions and queries executed locally without measurable overhead. Replication mechanism itself based on
23
-
logical decoding and earlier version of pglogical extension provided for community by 2ndQuadrant team.
24
-
25
-
Cluster consisting of N nodes can continue to work while majority of initial nodes are alive and reachable by
26
-
other nodes. This is done by using 3 phase commit protocol and heartbeats for failure discovery. Node that is
27
-
brought back to cluster can be fast-forwaded to actual state automatically in case when transactions log still
28
-
exists since the time when node was excluded from cluster (this depends on checkpoint configuration in postgres).
29
-
30
-
Read more about internals on [architecture](doc/architecture.md) page.
31
-
32
-
33
-
34
-
## Installation
35
-
36
-
Multi-master consist of patched version of postgres and extension mmts, that provides most of functionality, but
37
-
doesn't requiere changes to postgres core. To run multimaster one need to install postgres and several extensions
38
-
to all nodes in cluster.
39
-
40
-
### Sources
41
-
42
-
Ensure that following prerequisites are installed:
43
-
44
-
for Debian based linux:
45
-
46
-
```
47
-
apt-get install -y git make gcc libreadline-dev bison flex zlib1g-dev
# comma-separated list of connection strings to neighbour nodes.
108
-
```
15
+
## Overview
109
16
110
-
2. Allow replication in `pg_hba.conf`.
17
+
Multi-master replicates same database to all nodes in cluster and allows writes to each node. Transaction isolation is enforced cluster-wide, so in case of concurrent updates on different nodes database will use the same conflict resolution rules (mvcc with repeatable read isolation level) as single node uses for concurrent backends and always stays in consistent state. Any writing transaction will write to all nodes, hence increasing commit latency for amount of time proportional to roundtrip between nodes nedded for synchronization. Read only transactions and queries executed locally without measurable overhead. Replication mechanism itself based on logical decoding and earlier version of pglogical extension provided for community by 2ndQuadrant team.
111
18
112
-
Read description of all configuration params at [configuration](doc/configuration.md)
19
+
Cluster consisting of N nodes can continue to work while majority of initial nodes are alive and reachable by other nodes. This is done by using 3 phase commit protocol and heartbeats for failure discovery. Node that is brought back to cluster can be fast-forwaded to actual state automatically in case when transactions log still exists since the time when node was excluded from cluster (this depends on checkpoint configuration in postgres).
113
20
114
-
## Management
115
21
116
-
`create extension mmts;` to gain access to these functions:
22
+
## Documentation
117
23
118
-
* `mtm.get_nodes_state()` -- show status of nodes on cluster
119
-
* `mtm.get_cluster_state()` -- show whole cluster status
120
-
* `mtm.get_cluster_info()` -- print some debug info
121
-
* `mtm.make_table_local(relation regclass)` -- stop replication for a given table
24
+
1.[Administration](doc/administration.md)
25
+
1.[Installation](doc/administration.md)
26
+
1.[Setting up empty cluster](doc/administration.md)
27
+
1.[Setting up cluster from pre-existing database](doc/administration.md)
1.[Adding nodes to cluster](doc/administration.md)
31
+
1.[Excluding nodes from cluster](doc/administration.md)
32
+
1.[Architecture and internals](doc/architecture.md)
33
+
1.[List of configuration variables](doc/configuration.md)
34
+
1.[Built-in functions and views](doc/configuration.md)
122
35
123
-
Read description of all management functions at [functions](doc/functions.md)
124
36
37
+
## Tests
125
38
39
+
### Fault tolerance
126
40
127
-
## Tests
41
+
(Link to test/failure matrix)
128
42
129
43
### Performance
130
44
131
45
(Show TPC-C here on 3 nodes)
132
46
133
-
### Fault tolerance
134
-
135
-
(Link to test/failure matrix)
136
-
137
47
138
48
## Limitations
139
49
140
50
* Commit latency.
141
-
Current implementation of logical replication sends data to subscriber nodes only after local commit, so in case of
142
-
heavy-write transaction user will wait for transaction processing two times: on local node and on all other nodes
143
-
(simultaneosly). We have plans to address this issue in future.
51
+
Current implementation of logical replication sends data to subscriber nodes only after local commit, so in case of heavy-write transaction user will wait for transaction processing two times: on local node and on all other nodes (simultaneosly). We have plans to address this issue in future.
144
52
145
53
* DDL replication.
146
-
While data is replicated on logical level, DDL replicated by statements performing distributed commit with the same
147
-
statement. Some complex DDL scenarious including stored procedures and temp temp tables aren't working properly. We
148
-
are working right now on proving full compatibility with ordinary postgres. Currently we are passing 141 of 164
149
-
postgres regression tests.
54
+
While data is replicated on logical level, DDL replicated by statements performing distributed commit with the same statement. Some complex DDL scenarious including stored procedures and temp temp tables aren't working properly. We are working right now on proving full compatibility with ordinary postgres. Currently we are passing 141 of 164 postgres regression tests.
150
55
151
56
* Isolation level.
152
-
Multimaster currently support only _repeatable_ _read_ isolation level. This is stricter than default _read_ _commited_,
153
-
but also increases probability of serialization failure during commit. _Serializable_ level isn't supported yet.
57
+
Multimaster currently support only _repeatable__read_ isolation level. This is stricter than default _read__commited_, but also increases probability of serialization failure during commit. _Serializable_ level isn't supported yet.
1.[Adding nodes to cluster](doc/administration.md)
9
+
1.[Excluding nodes from cluster](doc/administration.md)
10
+
11
+
12
+
13
+
## Installation
14
+
15
+
Multi-master consist of patched version of postgres and extension mmts, that provides most of the functionality, but depends on several modifications to postgres core.
16
+
17
+
18
+
### Sources
19
+
20
+
Ensure that following prerequisites are installed:
21
+
22
+
for Debian based linux:
23
+
24
+
```
25
+
apt-get install -y git make gcc libreadline-dev bison flex zlib1g-dev
./configure --prefix=/path/to/install && make -j 4 install
43
+
cd contrib/mmts && make install
44
+
```
45
+
46
+
```./configure``` here is standard postgres autotools script, so it possible to specify [any options](https://www.postgresql.org/docs/9.6/static/install-procedure.html) available in postgres. Also please ensure that /path/to/install/bin is enlisted in ```PATH``` environment variable for current user:
47
+
48
+
```
49
+
export PATH=$PATH:/path/to/install/bin
50
+
```
51
+
52
+
53
+
### Docker
54
+
55
+
Directory contrib/mmts also includes docker-compose.yml that is capable of building multi-master and starting 3 node cluster listening on port 15432, 15433 and 15434.
56
+
57
+
```
58
+
cd contrib/mmts
59
+
docker-compose up
60
+
```
61
+
62
+
### PgPro packages
63
+
64
+
When things go more stable we will release prebuilt packages for major platforms.
65
+
66
+
67
+
68
+
## Configuration
69
+
70
+
After installing software on all cluster nodes we can configure our cluster. Here we describe how to set up multimaster consisting of 3 nodes with empty database. Suppose our nodes accesible via domain names ```node1```, ```node2``` and ```node3```. Perform following steps on each node (sequentially or in parallel – doesn't matter):
71
+
72
+
1. As with usual postgres first of all we need to initialize directiory where postgres will store it files:
73
+
```
74
+
initdb -D ./datadir
75
+
```
76
+
In that directory we are interested in files ```postgresql.conf``` and ```pg_hba.conf``` that are responsible for a general and security configuration consequently.
77
+
78
+
1. Create database that will be used with multimaster. This will require intermediate launch of postgres.
79
+
80
+
```
81
+
pg_ctl -D ./datadir -l ./pg.log start
82
+
createdb myuser -h localhost
83
+
createdb mydb -O myuser -h localhost
84
+
pg_ctl -D ./datadir -l ./pg.log stop
85
+
```
86
+
87
+
1. To be able to run multimaster we need following changes to ```postgresql.conf```:
88
+
89
+
```
90
+
### General postgres option to let multimaster work
91
+
92
+
wal_level = logical # multimaster is build on top of
93
+
# logical replication and will not work otherwise
94
+
max_connections = 100
95
+
max_prepared_transactions = 300 # all transactions are implicitly two-phase, so that's
96
+
# a good idea to set this equal to max_connections*N_nodes.
97
+
max_wal_senders = 10 # at least the number of nodes
98
+
max_replication_slots = 10 # at least the number of nodes
99
+
max_worker_processes = 250 # Each node has:
100
+
# N_nodes-1 receiver
101
+
# N_nodes-1 sender
102
+
# 1 mtm-sender
103
+
# 1 mtm-receiver
104
+
# Also transactions executed at neighbour nodes can cause spawn of
105
+
# background pool worker at our node. At max this will be equal to
106
+
# sum of max_connections on neighbour nodes.
107
+
108
+
### Multimaster-specific options
109
+
110
+
shared_preload_libraries = 'multimaster'
111
+
multimaster.max_nodes = 3 # cluster size
112
+
multimaster.node_id = 1 # the 1-based index of the node in the cluster
# comma-separated list of connection strings to neighbour nodes.
115
+
```
116
+
117
+
Full description of all configuration parameters available in section [configuration](doc/configuration.md). Depending on network environment and expected usage patterns one can want to tweak parameters.
118
+
119
+
1. Allow replication in `pg_hba.conf`:
120
+
121
+
```
122
+
host myuser all node1 trust
123
+
host myuser all node2 trust
124
+
host myuser all node3 trust
125
+
host replication all node1 trust
126
+
host replication all node2 trust
127
+
host replication all node3 trust
128
+
```
129
+
130
+
1. Finally start postgres:
131
+
132
+
```
133
+
pg_ctl -D ./datadir -l ./pg.log start
134
+
```
135
+
136
+
1. When postgres is started on all nodes you can connect to any node and create multimaster extention to get acces to monitoring functions:
137
+
```
138
+
psql -h node1
139
+
> CREATE EXTENSION multimaster;
140
+
```
141
+
142
+
To enshure that everything is working check multimaster view ```mtm.get_cluster_state()```:
143
+
144
+
```
145
+
> select * from mtm.get_cluster_state();
146
+
```
147
+
148
+
Check that liveNodes in this view is equal to allNodes.
149
+
150
+
151
+
## Setting up cluster from pre-existing database
152
+
## Tuning configuration params
153
+
## Monitoring
154
+
155
+
* `mtm.get_nodes_state()` -- show status of nodes on cluster
156
+
* `mtm.get_cluster_state()` -- show whole cluster status
157
+
* `mtm.get_cluster_info()` -- print some debug info
158
+
* `mtm.make_table_local(relation regclass)` -- stop replication for a given table
159
+
160
+
Read description of all management functions at [functions](doc/functions.md)
0 commit comments