You are hereZimbra Failing to Re-index a Mailbox

Zimbra Failing to Re-index a Mailbox


By steve - Posted on 18 November 2009

I was getting a fatal error affecting a Zimbra server running 5.0.2. A user with a large mailbox (10gb, 38000 messages) was having problems, so the first thing I wanted to do was re-index their mailbox. This can fix a multitude of sins, but it can also take awhile and it is a very bad thing to cancel.

Anyway, I tried to re-index the mailbox and it would process for awhile and get to about message 2724, then all of a sudden it would start failing to index certain messages, then really crap out. Then it would actually take the Zimbra server down and I would have to restart it each time!

It seemed to be message specific in that mailbox (failed at the same point each time), so I rooted around in the /opt/zimbra/log folder and checked out the logs. I found the message number that it was stopping on, then found it in the message store for the user and renamed it. (The message number in the log is the prefix for the msg file that is stored in the user's store in /opt/zimbra/store/0/##/
where ## is the user's internal number.)

That allowed the reindexing to get past that message, but then it died just a little later.

After a lot of research and turning the problem upside down, I finally figured some things out...

The server is running Ubuntu 6.06 LTS and Zimbra 5.0.2, so your mileage may vary. The system was also upgraded from previous releases, so, again, your mileage may vary.

To make sure it wasn't a corrupt index, I followed the directions here to delete the index files for that user. That's handy to know, but didn't help.

You see, the thing is, Zimbra is (and tomcat is as well) pretty greedy about opening files. This message in the log was finally a little helpful was:

java.net.SocketException: Too many open files

I wasn't sure if that was the symptom or the cause of the server crashing, but it looked promising.

Here's what I did to fix it:

1. Check the file limits as set by the kernel:

cat /proc/sys/fs/file-max
205174

Mine reported 205174, so that was OK.

2. Check the limits in the limits.conf file:

cat /etc/security/limits.conf
...
# End of file
zimbra soft nofile 524288
zimbra hard nofile 524288

So that looked good.

3. Check what the user zimbra has for limits:
su - zimbra
ulimits -n
1024

That's bad. That means the user zimbra is limited to 1024 files.

4. Fix the pam settings that aren't being read in:
nano /etc/pam.d/common-session
and add this line to the end of the file:
session required pam_limits.so

5. Make sure the zimbra user's limits are correct:
su - zimbra
ulimits -n
524288

6. Restart Zimbra:
/etc/init.d/zimbra restart

7. Reindex the user's mailbox using the Admin GUI or this command line:
su - zimbra
zmprov rim user.name@domain.tdl start

and check it using

zmprov rim user.name@domain.tdl status

Did this help you? You can help me!


Did you find this information helpful? You can help me back by linking to this page, purchasing from my sponsors, or posting a comment!


+One me on Google:


Follow me on twitter: http://twitter.com/mojocode







Comments



Affiliation Badges