Partitioned objects are replicated to multiple nodes based on the node list in their partition definition. Objects that have been replicated to one or more nodes are highly available and are available to the application following a node failure.
Replication can be synchronous or asynchronous on a per-node basis
in a partition. A mix of synchronous and asynchronous replication within
the same partition is supported. For example in Figure 7.1, “Partition definitions”, partition One is defined
to use synchronous replication to node B and
asynchronous replication to node C.
Synchronous replication guarantees that all replica nodes are updated in the same transaction in which the replicated object was modified. There can be no loss of data. However, the latency to update all of the replica nodes is part of the initiating transaction. By default, synchronous replication uses the deferred write protocol described in the section called “Deferred Write Protocol”.
Asynchronous replication guarantees that any modified objects are queued in a separate transaction. The object queue is per node and is maintained on the same node on which the modification occurred. Modified objects are updated on the replica nodes in the same order in which the modification occurred in the original transaction. The advantage of asynchronous replication is that it removes the update latency from the initiating transaction. However, there is potential for data loss if a failure occurs on the initiating node before the queued modifications have been replicated.
Figure 7.5, “Asynchronous replication” shows asynchronous replication behavior when a modification is made on the active node for a partition. The following steps are taken in this diagram:
A transaction is started.
Replicated objects are modified on the active node.
The modified objects are transactionally queued on the active node.
The transaction commits.
A separate transaction is started on the active node to replicate the objects to the target replica node.
The transaction is committed after all queued object modifications are replicated to the target node.
Because asynchronous replication is done in a separate transaction consistency errors can occur. When consistency errors are detected they are ignored, the replicated object is discarded, and a warning message is generated. These errors include:
Duplicate keys.
Duplicate object references caused by creates on an asynchronous replica.
Invalid object references caused by deletes on an asynchronous replica.
All other object modifications in the transaction are performed when consistency errors are detected.
Figure 7.6, “Replication protocol” provides more details on the differences between synchronous and asynchronous replication. The key things to notice are:
Synchronous modifications (creates, deletes, and updates) are always used when updating the active node and any synchronous replica nodes.
Modifications to asynchronous replica nodes are always done from the active node, this is true even for modifications done on asynchronous replica nodes.
![[Warning]](images/warning.png)
It is strongly recommend that no modifications be done on asynchronous replica nodes since there are no transactional integrity guarantees between when the modification occurs on the asynchronous replica and when it is reapplied from the active node.
Synchronous updates are always done from the node on which the modification occurred - this can be the active or a replica node.
These cases are shown in Figure 7.6, “Replication protocol”. These steps are shown in the
diagram for a partition P with the specified node
list:
A transaction is started on node
C- a replica node.A replicated object in partition P is modified on node
C.When the transaction is committed on node
C, the update is synchronously done on nodeA(the active node) and nodeB(a synchronous replica).Node
A(the active node) queues the update for nodeD- an asynchronous replica node.A new transaction is started on node
Aand the update is applied to nodeD.
If an I/O error is detected attempting to send creates, updates, or deletes to a replica node, an error is logged on the node initiating the replication and the object modifications for the replica node are discarded and the replica node is removed from the node list for the partition. These errors include:
the replica node has not been discovered yet
the replica node is down
an error occurred while sending the modifications
The replica node will be re-synchronized with the active node when the replica node is restored (see the section called “Restore”).