The two builds
Each game is installed twice on the same machine, in separate directories with separate save locations. The first installation stays clean for the life of the review: no mods, no configuration edits, no community patches, and updates applied only when the review is complete so that the shipped score describes a single known version.
The second installation carries the community's consensus load order. That phrase is doing real work and the next section defines it.
What counts as a consensus load order
Not our selection. If we chose the mods, the second score would measure our taste rather than the community's achievement, and comparing the two numbers would be meaningless.
A load order qualifies when it is recommended in at least three independent places — a pinned forum thread, a widely followed setup guide, a maintained collection — and where those sources agree on the core of the list. Mods that appear in some recommendations and not others are included only if they appear in a majority.
Where no consensus exists, the game does not receive a second score. Four games in the file are in this position: they have mods, but no agreed way of assembling them, and inventing one ourselves would defeat the purpose.
| Requirement | Threshold |
|---|---|
| Hours on shipped build | 40 minimum |
| Additional hours on modded build | 20 minimum |
| Independent sources agreeing on load order | 3 |
| Mods installed individually and verified | All |
| Updates tracked for breakage survey | All published |
The modding grade
Separate from both scores, because a game can be improved enormously by mods and still be hostile to the people making them. The grade answers one question: how well does this game support being changed?
Four things are weighed. Whether formats are readable — text configuration and documented binary formats score well, encryption scores nothing. Whether the studio publishes a schema or documentation. Whether an official loader or editor exists, and whether it has the same capability as the internal tools. And the breakage record, which is the heaviest single factor, taken from the update survey.
| Grade | Means |
|---|---|
| A+ | Documented, versioned interface, official tools at parity, no breakage |
| A | Readable formats, additive updates, little or no breakage, tools unofficial |
| B | Workable but undocumented, occasional breakage |
| C | Frequent breakage, or a hard ceiling on what can be reached |
| D | Actively difficult, most content types closed |
| F | Modification prevented by design |
Order of writing
The shipped review is written and finalised before the second installation is created. This is the single most important rule here, because knowing that a mod fixes something makes it very hard to judge honestly how badly the original was broken.
Once the shipped text is finished it is not revised. If the modded playthrough reveals that we misjudged something about the base game, that is noted in the second half of the review rather than corrected in the first.
The shipped score is finalised before a single mod is installed. Anything else would produce a review of our own hindsight.
Hardware and conditions
One machine, unchanged for the life of this publication, specified in the middle of the current mainstream rather than at the top. Both installations run on the same drive. Performance observations are relative between the two builds and are not presented as absolute figures, because a single configuration cannot support that claim.
All games are bought at retail with the publication's own money. No review copies are accepted, no early access is used for scoring, and no game is scored while under an embargo.
What this method cannot tell you
Four honest limits.
It cannot tell you what a load order will do on your machine, because mod interactions are sensitive to configuration in ways one test system cannot cover. It cannot tell you whether a load order will still work next month, only what its breakage history has been. It cannot account for the community work that was never attempted because the target was unstable. And it systematically favours games with large communities, because a game with six mods cannot produce a meaningful second score however good those six mods are.
Corrections
Where a published figure is wrong it is struck through rather than deleted, with the replacement beside it and the date of the change. Two conclusions in the first survey have been reversed this way and both reversals are described on that page rather than quietly absorbed.
Summary
Two installations, forty hours minimum on the shipped build and twenty further on the community's own consensus load order. Shipped review finalised before any mod is installed. Modding grade weighs formats, documentation, tooling and breakage record, with breakage weighted heaviest. Everything bought at retail on one unchanged machine.