Port389 - Recent changes [en]

Separate Conflict and Tombstone Entry

Nhosoi — Thu, 03 May 2012 16:34:18 GMT

Related Ticket/Bug:

← Older revision		Revision as of 16:34, 3 May 2012
Line 165:		Line 165:
	== Related Ticket/Bug ==		== Related Ticket/Bug ==
	Ticket #160 -- Make replication plugin put conflicts and tombstones in a special suffix<br>		Ticket #160 -- Make replication plugin put conflicts and tombstones in a special suffix<br>
-	Bug 747701 - [RFE] Make replication plugin put conflicts in a special suffix	+	Bug 695797 - Invalid host record created during client enrollment with failover<br>
		+	Bug 747701 - [RFE] Make replication plugin put conflicts in a special suffix<br>
		+	Bug 772294 - Replication conflics resolution

Use OpenLDAP Clients In 389

Rmeggins — Thu, 03 May 2012 14:30:08 GMT

Crypto:

← Older revision		Revision as of 14:30, 3 May 2012
(One intermediate revision not shown)
Line 187:		Line 187:
	** certfile is used for the name of the cert to use (for server or for client auth)		** certfile is used for the name of the cert to use (for server or for client auth)
	** keyfile is used for the key/cert db pin text (however, openldap only supports unlocked keys)		** keyfile is used for the key/cert db pin text (however, openldap only supports unlocked keys)
-	* ~~No PEM support yet - still need to integrate support for the PEM PKCS11 library~~	+	* more info [http://www.openldap.org/faq/data/cache/1514.html http://www.openldap.org/faq/data/cache/1514.html]
-	* No multi-init support yet - NSS upstream is working on this, so for now OpenLDAP is not really suitable for use in apps like nss_ldap	+
-		+
-	~~== Password hashing ==~~	+

Roadmap

Rmeggins — Wed, 02 May 2012 18:05:09 GMT

Version 1.2.11:

← Older revision		Revision as of 18:05, 2 May 2012
Line 12:		Line 12:
	* Support for Berkeley DB version 5		* Support for Berkeley DB version 5
	* Improved support for transactions and backend transaction plugins		* Improved support for transactions and backend transaction plugins
		+	** DNA and USN use backend transaction plugins
		+	** all plugins except replication, cos, roles, and views use backend transaction plugins
	* Allow internal operations triggered by external operations (referential integrity, memberof, etc.) to use modifiersName/creatorsName from original external operation		* Allow internal operations triggered by external operations (referential integrity, memberof, etc.) to use modifiersName/creatorsName from original external operation
	** Added a new config attribute '''nsslapd-plugin-binddn-tracking''' to cn=config. It is off by default. If set to on, the server will update a set of new operational attributes: '''internalModifiersname''' & '''internalCreatorsname'''. These attributes will store the DN of the plugin that made the update, while modifiersname/creatorsname will now be the bind dn that initiated the original external operation.		** Added a new config attribute '''nsslapd-plugin-binddn-tracking''' to cn=config. It is off by default. If set to on, the server will update a set of new operational attributes: '''internalModifiersname''' & '''internalCreatorsname'''. These attributes will store the DN of the plugin that made the update, while modifiersname/creatorsname will now be the bind dn that initiated the original external operation.
	* Windows Sync API version 2 [[Windows_Sync_Plugin_API]]		* Windows Sync API version 2 [[Windows_Sync_Plugin_API]]
		+	* Ability to disable replication agreements
		+	** '''nsds5ReplicaEnabled''' - if this is set to "off" in the replication agreement entry, the agreement will be disabled
		+	* CLEANRUV improvements [[Howto:CLEANRUV]]
		+	** New CLEANALLRUV and RELEASERUV tasks
		+	* Disk monitoring [[Disk_Monitoring]]
		+	** Can enable monitoring of disk usage, particularly for databases and log files, with warnings upon reaching certain thresholds of disk usage
		+	* Allow setup to work with IPv6

	=== Version 1.3 ===		=== Version 1.3 ===

History

Rmeggins — Wed, 02 May 2012 18:03:30 GMT

Version 1.2.9:

← Older revision		Revision as of 18:03, 2 May 2012
Line 185:		Line 185:
	** [[Auto_Membership_Design]]		** [[Auto_Membership_Design]]
	* More Coverity fixes		* More Coverity fixes
		+	=== Version 1.2.10 ===
		+	* Account Usability Control support
		+	* database transaction pre/post plugins
		+	** Changelog writes use main database transaction
		+	* native systemd support
		+	* slapi_rwlock support replaces direct NSPR PR_RWLock support

	== Initial Release ==		== Initial Release ==
	The initial release of Fedora Directory Server (version 7.1) was 6/1/2005. This included the source code to the complete Directory Server engine. It also included pre-built binaries (on selected platforms) for the admin server daemon and the console administration front-end, but not the source code for those. The entire product was open sourced on 12/1/2005 as Fedora Directory Server version 1.0 - see [[Release_Notes]] for more information about that release, which uses Apache as the admin server daemon. This was a week short of a year since Red Hat acquired the Directory Server from AOL, fulfilling the promise Red Hat made when it acquired the code to open source it within the year.		The initial release of Fedora Directory Server (version 7.1) was 6/1/2005. This included the source code to the complete Directory Server engine. It also included pre-built binaries (on selected platforms) for the admin server daemon and the console administration front-end, but not the source code for those. The entire product was open sourced on 12/1/2005 as Fedora Directory Server version 1.0 - see [[Release_Notes]] for more information about that release, which uses Apache as the admin server daemon. This was a week short of a year since Red Hat acquired the Directory Server from AOL, fulfilling the promise Red Hat made when it acquired the code to open source it within the year.

Roadmap

Rmeggins — Wed, 02 May 2012 18:03:07 GMT

Next Steps:

← Older revision		Revision as of 18:03, 2 May 2012
Line 6:		Line 6:
	== Next Steps ==		== Next Steps ==

-	~~=== Version 1.2.10 ===~~
-	* Account Usability Control support
-	* database transaction pre/post plugins
-	** Changelog writes use main database transaction
-	* native systemd support
-	* slapi_rwlock support replaces direct NSPR PR_RWLock support
	=== Version 1.2.11 ===		=== Version 1.2.11 ===
	* Support multiple Simple Paged Result searches in a single connection [[Simple_Paged_Results_Design]]		* Support multiple Simple Paged Result searches in a single connection [[Simple_Paged_Results_Design]]

Release Notes

Rmeggins — Wed, 02 May 2012 17:46:47 GMT

← Older revision		Revision as of 17:46, 2 May 2012
Line 1:		Line 1:
	__NOTOC__		__NOTOC__
		+	== 389 Directory Server 1.2.11.1 Testing - May 3, 2012 ==
		+	389-ds-base version 1.2.11.1 is released to Testing. This release contains some new features and many bug fixes.
		+
		+	The new packages and versions are:
		+	* 389-ds-base 1.2.11.1
		+
		+	NOTE: '''1.2.11 will not be available for Fedora 16 or earlier, nor for EL6 or earlier''' 1.2.11 will only be available for Fedora 17 and later. We are trying to stabilize current, stable releases - upgrades to 1.2.11 will disrupt stability.
		+
		+	=== Installation ===
		+	See [[Download]] for information about setting up your yum repositories. Then use '''yum install 389-ds'''
		+	yum install 389-ds
		+	After install completes, run '''setup-ds-admin.pl''' to set up your directory server.
		+	setup-ds-admin.pl
		+	=== Upgrade ===
		+	See [[Download]] for information about setting up your yum repositories. Then use '''yum upgrade'''
		+	yum upgrade
		+	After upgrade completes, run '''setup-ds-admin.pl -u''' to update your directory server/admin server/console information.
		+	setup-ds-admin.pl -u
		+
		+	See [[Download]] for more information about setting up yum access.
		+
		+	See [[Install_Guide]] for more information about the initial installation, setup, and upgrade.
		+
		+	See [[Source]] for information about source tarballs and SCM (git) access.
		+
		+	'''EL6''' - see [[Download]] for EL6 instructions
		+
		+	=== Your Feedback is Important! ===
		+	The best way to provide feedback is via the Fedora Update system.
		+
		+	* Go to https://admin.fedoraproject.org/updates
		+	* In the Search box in the upper right hand corner, type in the name of the package
		+	* In the list, find the version and release you are using (if you're not sure, use rpm -qi <package name> on your system) and click on the release
		+	* On the page for the update, scroll down to "Add a comment" and provide your input
		+
		+	Or just send us an email to 389-users@lists.fedoraproject.org
		+
		+	If you find a bug, or would like to see a new feature, use the 389 Trac - [https://fedorahosted.org/389 https://fedorahosted.org/389]
		+
		+	=== Notes ===
		+	NOTE: '''1.2.11 will not be available for Fedora 16 or earlier, nor for EL6 or earlier''' 1.2.11 will only be available for Fedora 17 and later. We are trying to stabilize current, stable releases - upgrades to 1.2.11 will disrupt stability.
		+
		+	=== New features in 1.2.11 ===
		+	* Transaction improvements
		+	** DNA and USN use backend transaction plugins
		+	** all plugins except replication, cos, roles, and views use backend transaction plugins
		+	* Track when password was last modified
		+	** '''passwordTrackUpdateTime''' - if this is set to "on", the server will keep track of the last password update time
		+	** '''pwdUpdateTime''' - new operational attribute in entries holds the last password update time if passwordTrackUpdateTime is enabled
		+	* Ability to disable replication agreements
		+	** '''nsds5ReplicaEnabled''' - if this is set to "off" in the replication agreement entry, the agreement will be disabled
		+	* CLEANRUV improvements [[Howto:CLEANRUV]]
		+	** New CLEANALLRUV and RELEASERUV tasks
		+	* Disk monitoring [[Disk_Monitoring]]
		+	** Can enable monitoring of disk usage, particularly for databases and log files, with warnings upon reaching certain thresholds of disk usage
		+	* Allow setup to work with IPv6
		+	* Additional winsync callback hooks [[Windows_Sync_Plugin_API]] - add post add/mod and AD add callback hooks
		+	* creatorsName and modifersName keep track of original bind DN for triggered plugin operations
		+	** '''nsslapd-plugin-binddn-tracking''' - if this is set to "on" in cn=config, the server will keep track of the original bind DN
		+	** '''internalModifiersname''' and '''internalCreatorsname''' - this will be the DN of the plugin that triggered the add/update operation
		+	* support for Berkeley DB version 5
		+	* support for multiple Simple Paged Searches per connection
		+	* support for SASL/PLAIN binds
		+
		+	=== Tickets for 1.2.10 ===
		+	These are all of the tickets targeted for 1.2.10. Tickets with Status '''closed''' and Resolution '''fixed''' have been fixed in this release.
		+	* [https://fedorahosted.org/389/report/12 389 1.2.10 Tickets]
		+
		+	=== Download, Install, Setup ===
		+	The [[Download]] page has information about how to get the binaries. The [[Install_Guide]] has information about installation and setup.
		+
	== 389 Directory Server 1.2.10.7 - April 30, 2012 ==		== 389 Directory Server 1.2.10.7 - April 30, 2012 ==
	389-ds-base version 1.2.10.7 is released to Stable. This release fixes some crashing issues seen with 1.2.10.4 and later.		389-ds-base version 1.2.10.7 is released to Stable. This release fixes some crashing issues seen with 1.2.10.4 and later.

Disk Monitoring

Rmeggins — Wed, 02 May 2012 17:35:31 GMT

Created page with '= Disk Space Monitoring = == Purpose == If available disk space gets critical this feature will shutdown the slapd process. This is to prevent the slapd process from crashing, ...'

New page

= Disk Space Monitoring =

== Purpose ==
If available disk space gets critical this feature will shutdown the slapd process. This is to prevent the slapd process from crashing, and possibly corrupting the db.

== New Config Options (cn=config) ==
* nsslapd-disk-monitioring: on|off (off by default)
* nsslapd-disk-monitoring-threshold: long (default is 2mb, value is in bytes)
* nsslapd-disk-monitoring-grace-period: int (60 minutes by default)
* nsslapd-disk-monitoring-preserve-logging: on|off (off by default)
* nsslapd-disk-monitoring-logging_critical: on|off (off by default)

== Details ==
At startup we create a new thread that will wake up every 10 seconds to check the available disk space on the disks/mount points that the DS is using. By default we only check the config, txn log directory, and db directories. If logging is critical to the deployment, and "nsslapd-disk-monitoring-logging-critical" is set, then we will also check the disks used for logging. Meaning, that if a log directory gets full, we will also shutdown the slapd process.

Once the available disk space on any of the disks get below the threshold we start taking action. If verbose error logging is enabled we turn it off. If logging is not critical, and we also don't want to preserve the logging, then we disable access & audit logging. On the next pass, if we continue to lose disk space, then we will delete all the rotated logs.

We will keep logging messages if the disk space continues to drop. If we get to the halfway point of the threshold, we go into shutdown mode.

Once in shutdown mode, we log another message warning that we will shutdown the process if disk space is not freed up within the grace period. If disk is freed up above the threshold, then we abort the shutdown, and enable any logging that we disabled. If it is not, then we shut slapd down.

Note - if at any point we get below 4k, we immediately shutdown.

== Config Details ==
* nsslapd-disk-monitioring: - Enables disk space monitoring
* nsslapd-disk-monitoring-threshold - The available disk space, in bytes, that will trigger the process. Default is 2mb. If we get below half of the threshold, 1 mb in this case, then we enter the shutdown mode.
* nsslapd-disk-monitoring-grace-period - How many minutes to wait to allow an admin to clean up disk space before shutting slapd down. The default is 60 minutes.
* nsslapd-disk-monitoring-preserve-logging - If this is set, we do not disable access & auditing logging, nor do we remove rotated logs. The only proactive step we take is to turn off verbose error logging.
* nsslapd-disk-monitoring-logging_critical - If this is set, we can go into shutdown mode if any of the log disks pass the halfway point of the threshold. We will also not disable logging or delete rotated logs when this is set.

Howto:CLEANRUV

Rmeggins — Wed, 02 May 2012 17:27:50 GMT

CLEANALLRUV and RELEASERUV:

← Older revision		Revision as of 17:27, 2 May 2012
(2 intermediate revisions not shown)
Line 1:		Line 1:
-	== ~~How~~ to use the ~~CLEANRUV task~~ ==	+	== When to use the tasks to cleanup the RUV ==

-	In a replicated environment, when you decommission a master, the meta-data for that master is still contained in the other servers. There ~~is a~~ special ~~task~~ you can use to remove this meta-data ~~- the CLEANRUV task~~. When you start up the server, you may get a warning like this in your errors log:	+	In a replicated environment, when you decommission a master, the meta-data for that master is still contained in the other servers. There are special tasks you can use to remove this meta-data. When you start up the server, you may get a warning like this in your errors log:

	[09/Sep/2011:09:03:43 -0600] NSMMReplicationPlugin - ruv_compare_ruv: RUV [changelog max RUV] does not		[09/Sep/2011:09:03:43 -0600] NSMMReplicationPlugin - ruv_compare_ruv: RUV [changelog max RUV] does not
Line 16:		Line 16:
	'(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))'		'(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))'
	There may be more than one replicated sub-suffix, so pay attention to which replica the message is complaining about.		There may be more than one replicated sub-suffix, so pay attention to which replica the message is complaining about.
		+
		+	== CLEANALLRUV and RELEASERUV ==
		+	How to remove a replica and its RUV's
		+
		+	1. On the Replica we want to remove, put database into read only mode to stop updates coming in.
		+	dn: cn=userRoot,cn=ldbm database,cn=plugins,cn=config
		+	changetype: modify
		+	replace: nsslapd-readonly
		+	nsslapd-readonly: on
		+	2. On the other valid replicas, delete the agreements to the replica we want to remove
		+	dn: cn=to repl_to_remove,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
		+	changetype: delete
		+	3. On the Replica we want to remove, delete all the agreements, and remove the replica entry
		+	dn: cn=to repl1,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
		+	changetype: delete
		+	...
		+
		+	dn: cn=to repl_last_one,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
		+	changetype: delete
		+
		+	dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
		+	changetype: delete
		+	4. Run CLEANALLRUV<rid> task. This operation will be "replicated" to all the servers inthe replication environment.
		+	dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
		+	changetype: modify
		+	replace: nsds5task
		+	nsds5task: CLEANALLRUV222
		+	5. Wait several minutes for all the replicas to have their RUV's cleaned. The amount of time is dependent on the amount of load. Confirm by searching on the tombstone entry on each replica:
		+	ldapsearch -xLLL -D "cn=directory manager" -w password -h localhost -b "dc=example,dc=com" \
		+	'(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' nsds50ruv
		+	6. Finally run the RELEASERUV<rid> task. This will allow a new replica, with the previously cleaned RID, to be added to the replication environment. It will also trigger changelog trimming.
		+	dn: cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
		+	changetype: modify
		+	replace: nsds5task
		+	nsds5task: RELEASERUV222
		+	7. Done
		+
		+	== CLEANRUV ==

	In order to run the CLEANRUV task, you will need to know		In order to run the CLEANRUV task, you will need to know

Release Notes

Rmeggins — Mon, 30 Apr 2012 19:33:04 GMT

389 Directory Server 1.2.10.4 - March 29, 2012:

← Older revision		Revision as of 19:33, 30 April 2012
Line 1:		Line 1:
	__NOTOC__		__NOTOC__
		+	== 389 Directory Server 1.2.10.7 - April 30, 2012 ==
		+	389-ds-base version 1.2.10.7 is released to Stable. This release fixes some crashing issues seen with 1.2.10.4 and later.
		+
		+	We have moved our ticket tracking system from the Red Hat Bugzilla [https://bugzilla.redhat.com/enter_bug.cgi?product=389 https://bugzilla.redhat.com/enter_bug.cgi?product=389] to our Fedora Hosted Trac [https://fedorahosted.org/389 https://fedorahosted.org/389]. All of the old 389 bugs have been copied to Trac. All new bugs, feature requests, and tasks should be entered in Trac.
		+
		+	'''WARNING''': If you are upgrading from a previous 1.2.6 release candidate, you will need to run fixfiles to fix some SELinux AVCs, or directory server will not start. See bug https://bugzilla.redhat.com/show_bug.cgi?id=622882
		+
		+	To fix, run this:
		+	fixfiles -R 389-ds-base restore
		+
		+	If you are upgrading from 1.2.5 or earlier, or a stable 1.2.6 or later, there is no problem.
		+
		+	'''WARNING''': If you are upgrading from a 1.2.6 alpha or release candidate, you will need to manually fix your entryrdn index files. See [[Subtree_Rename#warning:_upgrade_from_389_v1.2.6_.28a.3F.2C_rc1_.7E_rc6.29_to_v1.2.6_rc6_or_newer\|Subtree Rename Warning]] for more information. If you are upgrading from 1.2.5 or earlier, or a stable 1.2.6 or later, there is no problem.
		+
		+	The new packages and versions are:
		+	* 389-ds-base 1.2.10.7
		+
		+	=== Installation ===
		+	See [[Download]] for information about setting up your yum repositories. Then use '''yum install 389-ds'''
		+	yum install 389-ds
		+	After install completes, run '''setup-ds-admin.pl''' to set up your directory server.
		+	setup-ds-admin.pl
		+	=== Upgrade ===
		+	See [[Download]] for information about setting up your yum repositories. Then use '''yum upgrade'''
		+	yum upgrade
		+	After upgrade completes, run '''setup-ds-admin.pl -u''' to update your directory server/admin server/console information.
		+	setup-ds-admin.pl -u
		+
		+	See [[Download]] for more information about setting up yum access.
		+
		+	See [[Install_Guide]] for more information about the initial installation, setup, and upgrade.
		+
		+	See [[Source]] for information about source tarballs and SCM (git) access.
		+
		+	'''EL6''' - see [[Download]] for EL6 instructions
		+
		+	=== Your Feedback is Important! ===
		+	The best way to provide feedback is via the Fedora Update system.
		+
		+	* Go to https://admin.fedoraproject.org/updates
		+	* In the Search box in the upper right hand corner, type in the name of the package
		+	* In the list, find the version and release you are using (if you're not sure, use rpm -qi <package name> on your system) and click on the release
		+	* On the page for the update, scroll down to "Add a comment" and provide your input
		+
		+	Or just send us an email to 389-users@lists.fedoraproject.org
		+
		+	If you find a bug, or would like to see a new feature, use the 389 Trac - [https://fedorahosted.org/389 https://fedorahosted.org/389]
		+
		+	=== Notes ===
		+	NOTE: Fedora versions below 15 are no longer supported. If you are running Fedora 14 or earlier, you should upgrade.
		+
		+	NOTE: If you are using the console, after installing the updates, you must run '''setup-ds-admin.pl -u''' to refresh your console and admin server configuration with the new version information.
		+
		+	'''WARNING''': If you are upgrading from '''389 1.2.6 rcN (N < 6)''', simple upgrade won't work due to the entryrdn format change (See also [https://bugzilla.redhat.com/show_bug.cgi?id=616608 Bug 616608] - SIGBUS in RDN index reads on platforms with strict alignments ). Please follow the steps described in [http://directory.fedoraproject.org/wiki/Subtree_Rename#warning:_upgrade_from_389_v1.2.6_.28a.3F.2C_rc1_.7E_rc6.29_to_v1.2.6_rc6_or_newer this section].
		+
		+	'''WARNING''': If you are upgrading from a previous 1.2.6 release candidate, you will need to run fixfiles to fix some SELinux AVCs, or directory server will not start. See bug https://bugzilla.redhat.com/show_bug.cgi?id=622882
		+
		+	To fix, run this:
		+	fixfiles -R 389-ds-base restore
		+
		+	=== New features (up to and including 1.2.10.a8 - none after a8) ===
		+	* Account Usability Control support
		+	* database transaction pre/post plugins
		+	** Changelog writes use main database transaction
		+	** All plugins are transaction aware
		+	* native systemd support (Fedora 16 and later)
		+	* slapi_rwlock support replaces direct NSPR PR_RWLock support - solves some deadlock issues in certain cases
		+	* improved entryrdn index
		+	* Performance improvements
		+	** Reduce DN normalization
		+	** speed up searches that use large filters with large result sets
		+	* Automember and Managed Entry improvements
		+
		+	=== Tickets for 1.2.10 ===
		+	These are all of the tickets targeted for 1.2.10. Tickets with Status '''closed''' and Resolution '''fixed''' have been fixed in this release.
		+	* [https://fedorahosted.org/389/report/12 389 1.2.10 Tickets]
		+
		+	=== Download, Install, Setup ===
		+	The [[Download]] page has information about how to get the binaries. The [[Install_Guide]] has information about installation and setup.
		+
	== 389 Directory Server 1.2.10.4 - March 29, 2012 ==		== 389 Directory Server 1.2.10.4 - March 29, 2012 ==
	389-ds-base version 1.2.10.4 is released to Stable. This release contains a fix for a bug that causes the directory server to hang when using compare operations with virtual attributes.		389-ds-base version 1.2.10.4 is released to Stable. This release contains a fix for a bug that causes the directory server to hang when using compare operations with virtual attributes.

Source

Rmeggins — Mon, 30 Apr 2012 19:30:56 GMT

389 Directory Server 1.2.10:

← Older revision		Revision as of 19:30, 30 April 2012
Line 26:		Line 26:
	! SHA1SUM		! SHA1SUM
	\|-		\|-
-	\| [http://port389.org/sources/389-ds-base-1.2.10.6.tar.bz2 389-ds-base-1.2.10.6.tar.bz2]	+	\| [http://port389.org/sources/389-ds-base-1.2.10.7.tar.bz2 389-ds-base-1.2.10.7.tar.bz2]
-	\| ds.git \|\| 389-ds-base-1.2.10.6 \|\| [[Building]] \|\| ~~ba8e7bd8f4f3f6e3f5bb70508e9158c69decaa02~~	+	\| ds.git \|\| 389-ds-base-1.2.10.7 \|\| [[Building]] \|\| 46960c5ac6da03f999a72ad0d44c6f035d43aad5
	\|-		\|-
	\| [http://port389.org/sources/389-admin-1.1.29.tar.bz2 389-admin-1.1.29.tar.bz2]		\| [http://port389.org/sources/389-admin-1.1.29.tar.bz2 389-admin-1.1.29.tar.bz2]

Separate Conflict and Tombstone Entry

Nhosoi — Wed, 25 Apr 2012 19:43:24 GMT

New page

== Separate Conflict and Tombstone Entry ==
=== Entry format ===
==== conflict entry: ====
'''current format'''
dn: nsuniqueid=<UNIQID>+<RDN>,<PARENTDN>
<attr>: <value>
[...]
nsds5ReplConflict: namingConflict <DN>

'''proposing format (internal; Slapi_Entry)'''
dn: <RDN>,<PARENTDN>
objectclass: nsConflict
<attr>: <value>
[...]
nsdsEntryType: 1
nsds5ReplConflict: namingConflict <DN>

'''proposing format (external; search result/exported)'''
dn: nsuniqueid=<UNIQID>+<RDN>,<PARENTDN>
objectclass: nsConflict
<attr>: <value>
[...]
nsdsEntryType: 1
nsds5ReplConflict: namingConflict <DN>

==== tombstone entry: ====
'''current format'''
dn: nsuniqueid=<UNIQID>,<RDN>,<PARENTDN>
objectclass: nsTombstone
<attr>: <value>
[...]

'''proposing format (internal; Slapi_Entry)'''
dn: <RDN>,<PARENTDN>
objectclass: nsTombstone
<attr>: <value>
[...]
nsdsEntryType: 2

'''proposing format (external; search result/exported)'''
dn: nsuniqueid=<UNIQID>,<RDN>,<PARENTDN>
objectclass: nsTombstone
<attr>: <value>
[...]
nsdsEntryType: 2

==== ordinary entry (no changes): ====
dn: <RDN>,<PARENTDN>
<attr>: <value>
[...]

=== Goal ===
==== General ====
* Separate conflict entries and tombstone entries from ordinary entries.
* No change in the DN of the conflict entry and the tombstone entry in the DB (id2entry.db/entryrdn.db).
* The entries remain in id2entry.db with system attribute specifying the type:
** no type (or nsdsEntryType: 0) is for the ordinary entry
** nsdsEntryType: 1 for conflict
** nsdsEntryType: 2 for tombstone, for instance
<blockquote>
The system attribute nsdsEntryType is single-valued and integer. The motivation to have this system attribute is speeding up the entry type checking especially for the search-all "(objectclass=*)" case. We have the type info in the objectclass with the string value "nsConflict/nsTombstone". But checking the type using the string would be much slower than checking the integer value.
</blockquote>
* Add conflict_entryrdn for the conflict entry and tombstone_entryrdn for the tombstone entry. DN of the ordinary entries are managed in the entryrdn index; the conflict entries are in the conflict_entryrdn index / the tombstone entries are in the tombstone_entryrdn index.
* Add "objectclass: nsConflict" for the conflict entry as "objectclass: nsTombstone" for the tombstone entry.
* The internal entry does not include the special "nsuniqueid=<UNIQID>" in the DN.
* An entry could be a conflict and a tombstone entry at the same time. The entry would look like this and the RDN list is both in the conflict_entryrdn and tombstone_entryrdn.
<blockquote>
dn: <RDN>,<PARENTDN> 
objectclass: nsConflict 
objectclass: nsTombstone 
<attr>: <value> 
[...] 
nsdsEntrytype: 3 <== 1 | 2 
nsds5ReplConflict: namingConflict <DN>
</blockquote>
* We may not put the conflict / tombstone entry and its DN into the entry / DN cache not to waste the memory with the less prioritised objects, respectively.
==== Search ====
* Search requires "(objectclass=nsConflict)" / "(objectclass=nsTombstone)" in the filter to return the conflict / tombstone entry, respectively.
* DN in the returned entry has the following format to support multiple conflict/tombstone entries originated from the same DN.
** '''conflict entry''' nsuniqueid=<UNIQID>+<RDN>,<PARENTDN> 
** '''tombstone entry''' nsuniqueid=<UNIQID>,<RDN>,<PARENTDN>
* BaseDN in the search could include the special "nsuniqueid=<UNIQID>". In the current plan, both 2 cases return the same entry:
<blockquote>
'''case 1''' 
BaseDN: "nsuniqueid=<UNIQID>+<RDN>,<PARENTDN> 
Filter: (&(objectclass=nsConflict)(nsuniqueid=<UNIQID_N>)) 
'''case 2''' 
BaseDN: "<RDN>,<PARENTDN> 
Filter: (&(objectclass=nsConflict)(nsuniqueid=<UNIQID_N>)) 
'''Returned DN''' 
nsuniqueid=<UNIQID>+<RDN>,<PARENTDN> 
</blockquote>
* The same is true for the tombstone entry.
* Extra task for the search-all: search with "(objectclass=*)" needs to check the nsdsEntryType value and ignore the conflict/tombstone marked entries.
==== Add conflict entry ====
* Add the conflict entry with "objectclass: nsConflict" and "nsdsEntryType: 1" to id2entry.
* Add the DN (a list of RDNs) to the conflict_entryrdn index.
* The conflict_entryrdn could store the intermediate DNs which are not conflict entries. (the nodes could be marked as ordinary).
==== Delete entry/Add tombstone entry ====
* Modify the entry adding "objectclass: nsTombstone" (already implemented) and "nsdsEntryType: 2" in id2entry
* Delete the DN from the entryrdn.
* Add the DN (a list of RDNs) to the tombstone_entryrdn.
* The tombstone_entryrdn could store the intermediate DNs which are not tombstone entries. (the nodes could be marked as ordinary).
==== Export ====
If '-r' is added to the command line, conflict / tombstone entries are exported. The DN contains the special "nsuniqueid=<UNIQID>".
==== Import ====
Support the current ldif (no objectClass: nsConflict; no nsdsEntryType) as well as the new ldif.
==== Resurrection steps users take ====
* If an ordinary entry with the same DN exists in the DB, delete it.
* Modify the entry: removing "(objectclass=nsConflict)" / "(objectclass=nsTombstone)"
* Internally, the modify is translated to.
** removing "(objectclass=nsConflict)" / "(objectclass=nsTombstone)"
** removing nsdsEntryType (if not specified)
** removing nsds5ReplConflict (if not specified; just for the conflict entry)
** removing the DN from conflict_entryrdn / tombstone_entryrdn, respectively.
** adding the DN to entryrdn.
==== Upgrade ====
* Scan id2entry.
* If an entry's RDN is "nsuniqueid=<UNIQID>+<RDN>" and nsds5ReplConflict is in the attr,
** rename the RDN to <RDN>
** add "objectclass: nsConflict"
** remove the DN from entryrdn and add it to conflict_entryrdn
* If an entry's RDN is "nsuniqueid=<UNIQID>,<RDN>" and objectclass contains "nsTombstone",
** rename the RDN to <RDN>
** remove the DN from entryrdn and add it to tombstone_entryrdn
==== entrydn support ====
By setting "nsslapd-subtree-rename-switch: off" in cn=config, we could go back to the entrydn indexing. We have to introduce extra index files: conflict_entrydn and tombstone_entrydn, as well.
==== Internal format ====
To support the multiple conflict / tombstone entries having the same original DN, we allow entries having the same RDN and parentid in id2entry. (Note: the conflict / tombstone entries only)
dbscan -f id2entry.db output
id N
rdn: <RDN>
nsuniqueid: <UNIQID_N>
parentid: M
entryid: N

id N+1
rdn: <RDN>
nsuniqueid: <UNIQID_N+1>
parentid: M
entryid: N+1
Plus conflict_/tombstone_entryrdn should be able to store a list of entryids instead of one. (Again, this is only for the conflict/tombstone entries.)
==== Another approach to use the special suffix: The following change could be considered ====
'''conflict entry'''
dn: <RDN>,<PARENTDN>,<CONFLICTSUFFIX>
objectclass: nsConflict
<attr>: <value>
[...]
entrytype: 1

'''tombstone entry'''
dn: <RDN>,<PARENTDN>,<TOMBSTONESUFFIX>
objectclass: nsTombstone
<attr>: <value>
[...]
entrytype: 2

'''Comments on this approach'''
<blockquote>
* Adding the conflict / tombstone entry looks like a move in DIT, but it is not a move in terms of DN (since the original <SUFFIX> itself is not moved to <CONFLICTSUFFIX> or <TOMBSTONESUFFIX>, but a delete + add.
* ldbm_entryrdn code needs to have a knowledge about the special suffixes; currently, it only takes one "suffix".
* One entryrdn index file instead of 3 (entryrdn, conflict_entryrdn, tombstone_entryrdn).
* Search with -b "<CONFLICTSUFFIX>" / "<TOMBSTONESUFFIX>" is executed on multiple backends if there are multiple.
</blockquote>
== Related Ticket/Bug ==
Ticket #160 -- Make replication plugin put conflicts and tombstones in a special suffix 
Bug 747701 - [RFE] Make replication plugin put conflicts in a special suffix