Gerrit

From Wikitech

We use Gerrit for our code review system for Git. It supersedes the CodeReview extension.

Quick links

Details

Gerrit is installed on ytterbium in the prefix /var/lib/gerrit2. We appear to have no slave instances since formey was decommissioned. (Check this!) Replication for read-only is done to antimony (gitblit), gallium and lanthaum (continuous integration servers).

It is a Java daemon which listens for HTTP connections (port 8080) and SSH connections (port 29418). Apache on ytterbium proxies the relevant URLs on port 80 and 443 through to Gerrit. The SSH port provides a restricted shell for Git checkouts and administrative commands.

It uses the misc DB server (db1048), database name "reviewdb", for most storage. Git repos are on NFS and stored at /var/lib/gerrit2/review_site/git.

It uses the LDAP instance shared with Subversion and Wikimedia Labs, possibly on virt0.wikimedia.org, for authentication. Accounts under ou=people,dc=wikimedia,dc=org and groups under ou=groups,dc=wikimedia,dc=org are exposed to Gerrit. The Gerrit account name is the "cn" field in LDAP. In order to log in to Gerrit, a user needs to already have a Labs account. See Help:Access for the process of getting a Labs account.

Gerrit shells out to GitWeb, a Perl application which is installed using the stock package.

To find out what version of Gerrit is running, you can use either of the following two commands:

java -jar /var/lib/gerrit2/gerrit.war version
ssh -p 29418 gerrit.wikimedia.org gerrit version

It is 2.4.2-2-ge9a1970 at the time of writing.

Using git

This is help:git and on mediawiki.org.

Database install

Database installation is handled by Gerrit installation process, you simply need a user named 'gerrit2' with appropriate privileges. Make sure to set the charset of the database to utf8.

Replace <PASSWORD> below and apply:

 CREATE USER 'gerrit2'@'localhost' IDENTIFIED BY <PASSWORD>;
 CREATE DATABASE reviewdb;
 ALTER DATABASE reviewdb charset=utf8;
 GRANT ALL ON reviewdb.* TO 'gerrit2'@'localhost';
 FLUSH PRIVILEGES;


Upgrading Gerrit

  1. Back up the database (db1048.reviewdb) and git directories (/var/lib/gerrit2/review_site/git/). Just In Case.
  2. Download the latest .war file from Google Code to /var/lib/gerrit2/.
  3. Stop the gerrit daemon (/var/lib/gerrit2/review_site/bin/gerrit.sh stop)
  4. Start gerrit with the new *.war file using the "init" flag. For example: cd /var/lib/gerrit2 && java -jar gerrit-2.3.war init -d review_site
  5. The updater is going to ask you some questions, keep the current values. When it asks you if you want to copy the *.war file to the bin directory, say yes.
  6. Run puppet (this will restart gerrit)
  7. Profit!!

Note: You may need to restart gerrit after the upgrade is finished.

Getting the war

We build Gerrit + our plugins from master daily with Jenkins (plus when Chad feels like it). These builds are available from integration.mediawiki.org.

Generally speaking, the latest build is what we're running on labs. This will be the case for production soon too (right now we're using a custom 2.4.2 build).

Creating new repositories

see mw:Git/Creating new repositories

Deleting repositories

This isn't documented on mw.org since only Gerrit admins can do it and it's completely irrecoverable. Repeat, this action cannot be undone, you have been warned.

First, make sure that the repository that you want to delete is not used in CI (i.e. integration/config:/zuul/layout). If it is, file a bug against #Continuous-Integration, and either wait for the bug to be resolved or resolve it yourself by removing references to the repo from the CI configuration. Once the repository is no longer used in CI, you can use the following command to delete the repository from Gerrit itself:

ssh -p 29418 gerrit.wikimedia.org deleteproject delete [--force] [--yes-really-delete] -- foo/bar.git

After doing this, you also want:

  • To delete the repository from Github. For this you need a organization administrator in Github. Ask around in #wikimedia-operations for who can do it for you.
  • To delete the repository from git.wikimedia.org. You need ops or gitblit admin. Procedure is documented in Gitblit#Delete_a_repo
  • To delete the repository from phabricator.wikimedia.org. You needs ops or phab admin. Procedure is documented in Phabricator#Remove_a_repo

Indexing

We disabled indexing by search engines awhile back. Gitweb has awful performance, and the search engines were overloading manganese (mainly from requesting .tar urls from every commit they could dig for). We could possibly revisit this sometime in the future when we've got time to get actual stats (and we've replaced Gitweb with Gitblit).

Disabling account

If an account needs to be disabled for some reason, there's two steps to complete: remove the ssh keys and set the account to inactive. This action is limited to ldap/ops. The account_id can be found in the accounts table, if you don't know it.

$ ssh -p 29418 gerrit.wikimedia.org gerrit gsql
gerrit> update accounts set inactive = 'Y' where account_id = 1234;
gerrit> delete from account_ssh_keys where account_id = 1234;

Forcing Replication re-runs

When forcing a replication run through gerrit's replication plugin, stay clear of the --all switch, as it breaks renamed projects on github.

For example, both gerrit and github have a mediawiki repository. But github's mediawiki repository is actually mediawiki/core in gerrit. As gerrit also has a mediawiki repository, forcing replication on all projects would push gerrit's mediawiki repository onto github's mediawiki repository. Thereby, github's mediawiki repository falls over, and for examples the release tags from github's mediawiki are gone (cf. task T100409).

This currently (2015-05-28) affects us for at least for the following repos:

Gerrit's name Overloaded name on GitHub
mediawiki/core mediawiki
VisualEditor/VisualEditor VisualEditor
oojs/core oojs

If you by accident forced replication with --all, force replication of the effective gerrit repos to make up for it (i.e.: mediawiki/core, VisualEditor/VisualEditor, ...)

Fixing replication errors due to missing commits

If you see replication jobs in gerrit's queue that have an increasing retry count (>5); for example like here

 _________________________________________________________________
 christian@spencer // jobs: 0 // time: 15:00:42 // exit code: 0
 cwd: ~/tmp/gerrit-replication-fixup
 ssh gerrit.wikimedia.org gerrit show-queue -w | grep retry
 dea27401              12:59:49.618      (retry 676) push git@github.com:wikimedia/apps-ios-wikipedia

it typically means someone pushed a commit directly to github, by-passing gerrit and code-review. This is not a operational issue for gerrit as replication of other commits just continues and new commits make it into github. It's just that it keeps gerrit unnecessarily busy by retrying the same thing again and again and failing each time. And it pollutes gerrit logs.

This replication issue is fixed with upstream gerrit. Upgrading to >=2.9 should fix it for us. Since we're below 2.9, we have to fix manually; we have to manually bring the commit from github to gerrit.

First, we have to find the missing commit. To do so, log into the host thath is running gerrit (on 2015-07-07 that was ytterbium), and tail /var/lib/gerrit2/review_site/logs/error_log. You should see a line like

 [2015-07-07 12:48:44,586] ERROR com.googlesource.gerrit.plugins.replication.ReplicationQueue : Cannot replicate to git@github.com:wikimedia/apps-ios-wikipedia
 org.eclipse.jgit.errors.TransportException: git@github.com:wikimedia/apps-ios-wikipedia: Missing unknown 6a7595de888c8d04899cc58065fa8682eb844a39

. This means, one needs to bring commit 6a7595de888c8d04899cc58065fa8682eb844a39 from github to gerrit. So essentially:

  • Clone the github repo.
  • Push the commit to an unused branch in gerrit (refs/heads/sandbox/${user} ;-) ).
  • Wait for replication to happen.
  • Remove the unused branch from gerrit again.

Here's the same thing with more details:

  1. For the relevant project, grant Forge Author Identity, and Forge Committer Identity to the Project and Group Administrators group for the refs underneath refs/heads/sandbox/${username}. Otherwise, we cannot push another person's commit to our branch.
  2. _________________________________________________________________
    christian@spencer // jobs: 0 // time: 15:03:47 // exit code: 0
    cwd: ~/tmp/gerrit-replication-fixup
    git clone https://github.com/wikimedia/apps-ios-wikipedia.git
    Cloning into 'apps-ios-wikipedia'...
    remote: Counting objects: 17403, done.
    remote: Total 17403 (delta 0), reused 0 (delta 0), pack-reused 17403
    Receiving objects: 100% (17403/17403), 38.19 MiB | 938 KiB/s, done.
    Resolving deltas: 100% (11167/11167), done.
    Checking out files: 100% (2260/2260), done.
    
    _________________________________________________________________
    christian@spencer // jobs: 0 // time: 15:04:38 // exit code: 0
    cwd: ~/tmp/gerrit-replication-fixup
    cd apps-ios-wikipedia
    
    _________________________________________________________________
    christian@spencer // jobs: 0 // time: 15:04:40 // exit code: 0
    cwd: ~/tmp/gerrit-replication-fixup/apps-ios-wikipedia
    git branch replication-helper 6a7595de888c8d04899cc58065fa8682eb844a39
    
    _________________________________________________________________
    christian@spencer // jobs: 0 // time: 15:04:56 // exit code: 1
    cwd: ~/tmp/gerrit-replication-fixup/apps-ios-wikipedia
    git push ssh://gerrit.wikimedia.org/apps/ios/wikipedia replication-helper:refs/heads/sandbox/qchris/replication-helper
    Counting objects: 71, done.
    Delta compression using up to 8 threads.
    Compressing objects: 100% (20/20), done.
    Writing objects: 100% (44/44), 4.61 KiB, done.
    Total 44 (delta 32), reused 33 (delta 23)
    remote: Resolving deltas: 100% (32/32)
    remote: Processing changes: closed: 746, refs: 1, done    
    To ssh://gerrit.wikimedia.org/apps/ios/wikipedia
     * [new branch]      replication-helper -> sandbox/qchris/replication-helper
    
  3. Wait 1 minute, for replication to run the next try.
  4. Check if the replication job vanished from gerrit's queue (Re-run the show-queue command from the beginning of this section). If it did not vanish, the job is blocked on another commit. The error_log will show you the additional commit that you'll need to bring over from github to gerrit.
  5. Drop the sandbox/qchris/replication-helper again from gerrit:
    _________________________________________________________________
    christian@spencer // jobs: 0 // time: 15:15:36 // exit code: 1
    cwd: ~/tmp/gerrit-replication-fixup/apps-ios-wikipedia
    git push ssh://gerrit.wikimedia.org/apps/ios/wikipedia +:refs/heads/sandbox/qchris/replication-helper
    remote: Processing changes: refs: 1, done    
    To ssh://gerrit.wikimedia.org/apps/ios/wikipedia
     - [deleted]         sandbox/qchris/replication-helper
    
  6. Drop the identity forging grants from the project config in gerrit again.

See also