When I left Harvard, I had been told that until I graduated from another institution, I would be considered to be “on a leave of absence”. This meant that I could return at any time, since I was still a Harvard student. So I’ve been going through this past semester always slightly uncertain of my identity. I still don’t feel like a “real” MIT student, and part of that is because Harvard still thinks of me as one of their own. And so yesterday when I tried to log on to the Harvard College internal facebook (not to be confused with Facebook, although as I understand it there is some interesting history there), I was shocked to discover the following:
While it’s still true that I can go back to Harvard, this was the first time that I really had to think about the fact that I’ve left, and in doing so, I am a Harvard student no longer.
So I’ve been spending oodles of time over the past few days working on making anygit, an index of the world’s git repositories. We needed a highly scalable database to store all of the index information. After poking at the various possibilities, we settled on the ridiculously awesome MongoDB. This database is scalable to very large sizes, and it has useful features such as auto-sharding.
However, this is not a story about what MongoDB has. No, it is a story about what MongoDB lacks.
After spending many hours creating my software, I naturally wanted to test it. I spun up an Ubuntu Karmic VM on XVM and installed the mongodb package on it. My indexer seemed to work just fine, but after some time, I noticed a number of git objects that did not seem to have contain any index information. I spent several hours investigating these objects and double checking my code. Why were some blobs not remembering their filenames? Why did some trees refuse to record which commits they came from? Finally, I looked at the database log. And there I was confronted with many messages of the form:
Sat May 22 15:11:27 allocating new datafile /var/lib/mongodb/anygit.5, filling with zeroes...
Sat May 22 15:11:27 posix_fallocate failed: errno:28 No space left on device falling back
Sat May 22 15:11:27 Assertion: 10440:failure creating new datafile
...
Sat May 22 15:11:28 Caught Assertion in update , continuing
Sat May 22 15:11:28 update anygit.git_objects query: { _id: "36f171b3028a3460c1c51f9c1f5747a9a0d2d850" } exception massert:file allocation failure 697ms
Sat May 22 15:11:29 Assertion: 12520:file allocation failure
Wow. So apparently I was out of disk, and my client was neglecting to tell me that. (I’m not sure whether the server was reporting this to the client, but I suspect it was not.) Case closed, and everything else should be smooth from here, right? Wrong. I later ran into other resource issues, including OOM kills (MongoDB mmaps the database files, so it will consume all the memory you have. It seems as though sometimes your virtual memory manager isn’t smart enough to realize it can drop all of the mmap’ed files rather than OOM killing.) I also ran into an issue with the packaged version on 64 bit Lucid where MongoDB would allow inserts but silently ignore updates (including in the logs). I haven’t worked hard to reproduce this, so I could be entirely mistaken, however.