Move Replication Changelog


Background

Improved performance, robustness and consistency

By sharing the main DB’s dbEnv with the replication changelog, the above issues are solved.

Note: the version which supports merged changelog is v1.2.7 or newer

Implementation

_cl5AppInit (replication/cl5_api.c) retrieves dbEnv and pagesize from the backend plugin (back-ldbm) using the newly introduced slapi API slapi_back_get_info. The rest of the logic handling changelog db remains intact. By sharing the backend DB env, we could eliminate the following changelog db configuration parameters under cn=changelog5,cn=config:

nsslapd-cachesize    
nsslapd-cachememsize    
nsslapd-db-checkpoint-interval    
nsslapd-db-circular-logging    
nsslapd-db-logfile-size    
nsslapd-db-max-txn    
nsslapd-db-verbose    
nsslapd-db-debug    
nsslapd-db-trickle-percentage    
nsslapd-db-spin-count    
nsslapd-db-locks    

Note: the following parameters are still available.

nsslapd-changelogdir    
nsslapd-changelogmaxentries    
nsslapd-changelogmaxage    
nsslapd-changelogmaxconcurrentwrites    

Backup/Restore: db2bak and bak2db backs up and restores the changelogs in the changelog directory if the changelog entry (cn=changelog5,cn=config) as well as the value of nsslapd-changelogdir in the entry exist, the directory is copied to the backup directory under the directory .repl_changelog_backup. The changelog items are guaranteed to be synchronized with the main DB. Upgrade: an upgrade script calls db_checkpoint against the changelog db files, then remove the DB env files and transaction log files.

New APIs

int slapi_back_get_info(Slapi_Backend *be, int cmd, void **info);    
* Get backend info based upon cmd    
*    
* parameters    
* be Backend from which the information will be retrieved    
* cmd macro to specify the information type    
* info pointer to store the information    
* return values    
* 0 if the operation was successful    
* non-0 if the operation was not successful    
*    
* Implemented cmd:    
* BACK_INFO_DBENV - Get the dbenv    

int slapi_back_set_info(Slapi_Backend *be, int cmd, void *info);    
* Set info to backend based upon cmd    
*    
* parameters    
* be Backend to which the information will be set    
* cmd macro to specify the information type    
* info pointer to the information    
* parameters    
* 0 if the operation was successful    
* non-0 if the operation was not successful    
*    
* Warning: No cmd is defined yet.    

Test Scenario

(*) The changelog directory is backed up to the directory .repl_changelog_backup under the back up directory. For instance, if the back up dir is /var/lib/dirsrv/slapd-ID/bak/todaysbakup and changelogdir name is cldir, then it will be backed up in /var/lib/dirsrv/slapd-ID/bak/todaysbakup/.repl_changelog_backup/cldir. “.repl_changelog_backup” is chosen to avoid the conflict with the backend instance name (e.g., netscaperoot, userroot).

Backing up Changelog

The changelog db is now a part of the main backend database. The backing up utility db2bak[.pl]/Back Up on Console includes the changelog db’s in the backed up database. By design, changelog db stores the RUVs at the close time. When starting up the server, it reads the stored RUVs from the changelog, sets them in memory and deletes them from the changelog. That is, when the server is running, the changelog db does not contain the RUV information. If the changelog dbs which do not have the RUVs are backed up, the server with the restored changelog dbs from the backup behaves abnormally. To prevent the problem, this restriction has been introduced to the standalone utility db2bak. db2bak.pl and Console Back Up has no limitation.

db2bak.pl and Console Back up    
  -- the instance has no changelogs --> back up main backend db    
  -- the instance has changelogs    --> add RUVs to changelog; back up main backend db & changelog db; delete RUVs from changelog    
db2bak    
  -- the instance has no changelogs --> back up main backend db    
  -- the instance has changelogs    -- the server is down --> back up main backend db & changelog db, which stores the RUVs    
                                    -- the server is up   --> back up fails with this error:    
                                          [...] - db2archive: pre-backup-plugin failed (1).    
                                          [...] - ERROR: Standalone db2bak is not supported \    
                                          when a multisupplier replication enabled server is coexisting.    
                                          Please use db2bak.pl, instead.    

To back up the database while the server is up, task version of the back up utility (db2bak.pl or Console) needs to be used. The standalone utility db2bak can be used either when the server is down or when the the server is not a supplier.

Transactions

Now that the changelog shares the main database environment, it can use the main database transaction for its writes. This will make it so that the changelog can never be out of sync with the main database. It should also speed up writes - if all writes are committed in a single transaction, that’s a single call to fdatasync() instead of two.

Tasks

Related Bugs

Bug 633168 - Share backend dbEnv with the replication changelog Bug 669205 - db2bak: backed up changelog should include RUVs

Last modified on 1 March 2024