Part 1 of 5

Three Hundred Files

No Forum Software

Between roughly 2007 and 2016 I ran an online community for 3D artists. It started as 3dsmaxforum.com and later became digitalartsfront.com. The first thing to say about it is what isn't in it.

There is no vBulletin. No phpBB, no Simple Machines, no Invision, no MyBB. In 2007 any of those would have given me a forum in an afternoon, and several were free. What is there instead is 307 PHP files, written by hand, sitting on a MySQL schema of 62 tables.

Two packages did make it onto the server, and they are the exception that shows where the line was. A WordPress install sat under labs/ running a blog, on its own database of fourteen tables, and a stock phpBB sat in a database beside it serving a different forum. Neither touched this community: not its forum, not its gallery, not its challenges, not its ads. Where somebody else's software would have done the job, I used it. This site is the part I wrote.

This is the first of five posts about that codebase. The others go into the search engine, the chat daemons, the two scoring systems, and what the rebrand left behind.

What Was Actually in There

A forum, which is the part the domain name promises. Then, in the same codebase and the same schema:

An image gallery with portfolios, categories, and per-image comment threads. A tutorials section. A jobs board. Art challenges with entries and voting. Private messaging. Polls. A shoutbox. A live chat system. A search engine with its own index. A referral system. Medals and ranks. RSS feeds. A Facebook canvas application.

And an advertising platform. Not a Google AdSense tag, though those are in there too. A self-serve one, where an advertiser could buy a placement, upload creative, and get impression and click reporting, paid through PayPal, with subscription billing for recurring buys.

The database is 2.3 GB, and sixty-two tables carry all of it: forumBoards, challenges, ads, comments, accessLog and the rest, every name my own.

There is a second database on the same host that also has exactly sixty-two tables, which looks for a moment like the same engine running a second community. It isn't. Every table in it is prefixed phpbb_. Sixty-two is a coincidence, and the neighbouring database is a stock phpBB install serving something else entirely.

It's also the sharpest version of the question this post is supposed to answer. The software I said I didn't use was sitting on the same machine the whole time, so I knew exactly what I was turning down.

Why Not Just Install phpBB

In 2007 I wanted to write it more than I wanted to have it. That is a bad reason to build software at work and a defensible one for a side project.

There is a better answer underneath it, though, and the schema is the evidence. A forum install gives a forum. What this site turned into was a gallery with a forum attached, then a gallery and a forum with a challenge system attached, then all of that with an ad business on top. Every one of those needs to know about users, comments, images, and reputation, and every one of them wants to reach into the others.

The comment system shows it. There is one comments table, and it carries a commentType column. Forum replies, image comments, tutorial comments, challenge feedback, and replies to comments are all rows in it. Threading is a parentId pointing at another comment. That means a feature written for one part of the site works everywhere the moment it is written, and it means the search indexer can walk a comment tree without caring what the tree is hanging off.

Bolting a gallery onto phpBB in 2007 meant either writing a bridge and keeping two user tables in sync, or writing the gallery inside a plugin architecture designed for a forum. Both were common and both were miserable. Writing the whole thing meant one users table, one comments table, and no bridge.

That is the argument, and I still think it holds for this specific case. It does not generalize. The reason it worked here is that the surface never had to interoperate with anything else, and the only person who had to understand the whole model was me.

What It Cost

The parts of this codebase that are hardest to defend are the parts a forum install would have got right for free.

Every query in it is string-concatenated SQL. Some of it goes through mysql_real_escape_string and some of it does not. The database object has a query cache and per-request timing instrumentation, which is more attention than most sites of this size gave the problem, and it has no parameterized queries at all, because the API being used did not have them.

Passwords, sessions, and the account recovery flow are all hand-written. So is the moderation tooling. So is the rate limiting, such as it is. Each of those is a place where a mature forum package had already learned from a decade of other people's incidents, and this code had learned from none of them.

What Transfers

One table for a concept beats one table per feature. The single comments table with a type column is the decision that let this site keep growing sideways. Threading, moderation, search indexing, and notification were each written once. The cost is that the table has no referential integrity, because a parentId means something different depending on the type beside it, and nothing in the schema enforces the pairing.

Building it yourself is a reasonable answer to integration, not to features. Nothing in this codebase does forum posting better than phpBB did in 2007. What it does better is not have a seam down the middle of it. If the thing being built is one feature, install the package. If it is four features that all want the same user, the seam is the expensive part.

The next post is about the search engine, which is a table of words and a table of counts, and which decides that a forum thread and all of its replies are one document.