LDAP is a directory service we use to inventory the users, groups, passwords, (some) email forwards and machines.

Tutorial

The LDAP interface documentation is on db.torproject.org. See specifically the instructions on how to:

How-to

Troubleshooting changes@ failures

A common user question is that they are unable to change their SSH key. This can happen if their email client somehow has trouble sending a PGP signature correctly. Most often than not, this is because their email client does a line wrap or somehow corrupts the OpenPGP signature in the email.

A good place to start looking for such problems is the log files on the LDAP server (currently alberti). For example, this has a trace of all the emails received by the changes@ alias:

/srv/db.torproject.org/mail-logs/received.changes

A common problem is people using --clearsign instead of --sign when sending an SSH key. When that hapepns, many email clients (including Gmail) will word-wrap the SSH key after the comment, breaking the signature. For example, this might happen:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKxqYYEeus8dRXBHhLsp0SjH7ut2X8UM9hdXN=
wJIl89otcJ5qKoXj90K9hq8eBjG2KuAZtp0taGQHqzBOFK+sFm9/gIqvzzQ07Pn0xtkmg10Hunq=
vPKMj4gDFLIqTF0WSPA2E6L/TWaeVJ+IiGuE49j+0Ohd7UFDEquM1H/zno22vIEm/dxWLPWD9gG=
MmwBghvfK/dRyzSEDGlAVeWLzoIvVOG12/ANgic3TlftbhiLKTs52hy8Qhq/aQBqd0McaE4JGxe=
9k71OCg+0WHVS4q7HVdTUqT3VFFfz0kjDzYTYQQcHMqPHvYzZghxMVCmteNdJNwJmGSNPVaUeJG=
MumJ9
anarcat@curie

-----BEGIN PGP SIGNATURE-----
[...]
-----END PGP SIGNATURE-----

Using --sign --armor will work around this problem, as the original message will all be ascii-armored.

Restoring from backups

There's no special backup procedures for the LDAP server: it's backed up like everything else in the backup system.

To restore the OpenLDAP database, you need to head over the Bacula director, and enter the console:

ssh -tt bacula-director-01 bconsole

Then call the restore command and select 6: Select backup for a client before a specified time. Then pick the server (currently alberti.torproject.org) and a date. Then you need to "mark" the right files:

cd /var/lib/ldap
mark *
done

Then confirm the restore. The files will end up in /var/tmp/bacula-restores on the LDAP server.

The next step depends on whether this is a partial or total restore.

Partial restore

If you only need to access a specific field or user or part of the database, you can use slapcat to dump the database from the restored files even if the server is not running. You first need to "configure" a "fake" server in the restore directory. You will need to create two files under /var/tmp/bacula-restores:

  • /var/tmp/bacula-restores/etc/ldap/slapd.conf
  • /var/tmp/bacula-restores/etc/ldap/userdir-ldap-slapd.conf

They can be copied from /etc, with the following modificatiosn:

diff -ru /etc/ldap/slapd.conf etc/ldap/slapd.conf
--- /etc/ldap/slapd.conf    2011-10-30 15:43:43.000000000 +0000
+++ etc/ldap/slapd.conf 2019-11-25 19:48:57.106055596 +0000
@@ -17,10 +17,10 @@

 # Where the pid file is put. The init.d script
 # will not stop the server if you change this.
-pidfile         /var/run/slapd/slapd.pid
+pidfile         /var/tmp/bacula-restores/var/run/slapd/slapd.pid

 # List of arguments that were passed to the server
-argsfile        /var/run/slapd/slapd.args
+argsfile        /var/tmp/bacula-restores/var/run/slapd/slapd.args

 # Read slapd.conf(5) for possible values
 loglevel        none
@@ -57,4 +57,4 @@
 #backend       <other>

 # userdir-ldap
-include /etc/ldap/userdir-ldap-slapd.conf
+include /var/tmp/bacula-restores/etc/ldap/userdir-ldap-slapd.conf
diff -ru /etc/ldap/userdir-ldap-slapd.conf etc/ldap/userdir-ldap-slapd.conf
--- /etc/ldap/userdir-ldap-slapd.conf   2019-11-13 20:55:58.789411014 +0000
+++ etc/ldap/userdir-ldap-slapd.conf    2019-11-25 19:49:45.154197081 +0000
@@ -5,7 +5,7 @@
 suffix          "dc=torproject,dc=org"

 # Where the database file are physically stored
-directory       "/var/lib/ldap"
+directory       "/var/tmp/bacula-restores/var/lib/ldap"

 moduleload      accesslog
 overlay accesslog
@@ -123,7 +123,7 @@


 database hdb
-directory       "/var/lib/ldap-log"
+directory       "/var/tmp/bacula-restores/var/lib/ldap-log"
 suffix cn=log
 #
 sizelimit 10000

Then slapcat is able to read those files directly:

slapcat -f /var/tmp/bacula-restores/etc/ldap/slapd.conf -F /var/tmp/bacula-restores/etc/ldap

Copy-paste the stuff you need into ldapvi.

Full rollback

Untested procedure.

If you need to roll back the entire server to this version, you first need to stop the LDAP server:

service slapd stop

Then move the files into place (in /var/lib/ldap):

mv /var/lib/ldap{,.orig}
cp -R /var/tmp/bacula-restores/var/lib/ldap /var/lib/ldap
chown -R openldap:openldap /var/lib/ldap

And start the server again:

service slapd start

Listing members of a group

To tell which users are part of a given group (LDAP or otherwise), you can use the getent(1) command. For example, to see which users are part of the tordnsel group, you would call this command:

$ getent group tordnsel
tordnsel:x:1532:arlo,arma

In the above, arlo and arma are members of the tordnsel group. The fields in the output are in the format of the group(5) file.

Note that the group membership will vary according to the machie on which the command is run, as not all users are present everywhere.

Searching LDAP

This will load a text editor with a dump of all the users (useful to modify an existing user or add a new one):

ldapvi -ZZ --encoding=ASCII --ldap-conf -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org"

This will list all known hosts in LDAP:

ldapsearch -ZZ -vLxW -h db.torproject.org -D "uid=$USER,ou=users,dc=torproject,dc=org" -b "ou=hosts,dc=torproject,dc=org" '(objectclass=*)' | grep ^dn:

Modifying the schema

If you need to add, change or remove a field in the schema of the LDAP database, it is a different, and complex operation. You will only need to do this if you launch a new service that (say) requires a new password specifically for that service.

The schema is maintained in the userdir-ldap.git repository. It is stored in the userdir-ldap.schema file. Assuming the modified object is a user, you would need to edit the file in three places:

  1. as a comment, in the beginning, to allocate a new field, for example:

    @@ -113,6 +113,7 @@
     #   .45 - rebootPolicy
     #   .46 - totpSeed
     #   .47 - sshfpHostname
    +#   .48 - mailPassword
     #
     # .3 - experimental LDAP objectClasses
     #   .1 - debianDeveloper
    

This is purely informative, but it is important as it serves as a central allocation point for that numbering system. Also note that the entire schema lives under a branch of the Debian.org IANA OID allocation.

  1. create the actual attribute, somewhere next to a similar attribute or after the previous OID, in this case we created an attributed called mailPassword right after rtcPassword, since other passwords were also grouped there:

    attributetype ( 1.3.6.1.4.1.9586.100.4.2.48
           NAME 'mailPassword'
           DESC 'mail password for SMTP'
           EQUALITY octetStringMatch
           SYNTAX 1.3.6.1.4.1.1466.115.121.1.40 )
    
  2. finally, the new attribute needs to be added to the objectclass. in our example, the field was added alongside the other password fields in the debianAccount objectclass, which looked like this after the change:

    objectclass ( 1.3.6.1.4.1.9586.100.4.1.1
        NAME 'debianAccount'
        DESC 'Abstraction of an account with POSIX attributes and UTF8 support'
        SUP top AUXILIARY
        MUST ( cn $ uid $ uidNumber $ gidNumber )
        MAY ( userPassword $ loginShell $ gecos $ homeDirectory $ description $ mailDisableMessage $ sudoPassword $ webPassword $ rtcPassword $ mailPassword $ totpSeed ) )
    

Once that schema file is propagated to the LDAP server, this should automatically be loaded by slapd when it is restarted (see below). But the ACL for that field should also be modified. In our case, we had to add the mailPassword field to two ACLs:

--- a/userdir-ldap-slapd.conf.in
+++ b/userdir-ldap-slapd.conf.in
@@ -54,7 +54,7 @@ access to attrs=privateSub
        by * break

 # allow users write access to an explicit subset of their fields
-access to attrs=c,l,loginShell,ircNick,labeledURI,icqUIN,jabberJID,onVacation,birthDate,mailDisableMessage,gender,emailforward,mailCallout,mailGreylisting,mailRBL,mailRHSBL,mailWhitelist,mailContentInspectionAction,mailDefaultOptions,facsimileTelephoneNumber,telephoneNumber,postalAddress,postalCode,loginShell,onVacation,latitude,longitude,VoIP,userPassword,sudoPassword,webPassword,rtcPassword,bATVToken
+access to attrs=c,l,loginShell,ircNick,labeledURI,icqUIN,jabberJID,onVacation,birthDate,mailDisableMessage,gender,emailforward,mailCallout,mailGreylisting,mailRBL,mailRHSBL,mailWhitelist,mailContentInspectionAction,mailDefaultOptions,facsimileTelephoneNumber,telephoneNumber,postalAddress,postalCode,loginShell,onVacation,latitude,longitude,VoIP,userPassword,sudoPassword,webPassword,rtcPassword,mailPassword,bATVToken
        by self write
        by * break

@@ -64,7 +64,7 @@ access to attrs=c,l,loginShell,ircNick,labeledURI,icqUIN,jabberJID,onVacation,bi
 ##

 # allow authn/z by anyone
-access to attrs=userPassword,sudoPassword,webPassword,rtcPassword,bATVToken
+access to attrs=userPassword,sudoPassword,webPassword,rtcPassword,mailPassword,bATVToken
        by * compare

 # readable only by self

If those are the only required changes, it is acceptable to directly make those changes directly on the LDAP server, as long as the exact same changes are performed in the git repositoy.

It is preferable, however, to build and upload userdir-ldap as a Debian package instead.

Reference

LDAP is not accessible to the outside world, so you need to be behind the firewall. Once that's resolved, you can use ldapvi(1) or ldapsearch(1) to inspect the database. User documentation on that process is in accounts.

Design

The LDAP setup at Tor is based on the one from Debian.org. /etc/password and groups files are synchronized from the central LDAP server using the sshdist account, which means things keep working when LDAP is down. Most operations can be performed on the db.torproject.org site or by email.

DNS zone files are also managed (at least partly) in LDAP. This is automated through cron jobs, but if you're in a hurry, the zones get generated by ud-generate on alberti (as sshdist?) and replicate (?) on nevii with ud-replicate (as root?).