On Lens Reviews

I appreciate Erwin Puts' comments in his recent article "On Lens Reviews," and I think he's on to something. His clarification of the various approaches to lens reviewing is right on, and better articulated by him here than I've seen from other writers elsewhere.

He's got me pegged, for one thing, when he says that my approach "takes the whole imaging and viewing chain as an integral process and reviews a lens for its impact on the presentation of the scene as fixed on a print and viewed by an observer." I couldn't have said it better myself, and it's very true—the proof for me is the print, and what you can't see in the print doesn't count for very much as far as I'm concerned.

In other words, I'm an "eyeballer," as Phil Davis used to call me (not very approvingly)—although I'll register my usual protest, which is to say that I think I'm a good eyeballer.

But the empirical, practical approach is in some ways not very descriptive. In fact, I don't even like to call my lens reviews "tests"—I prefer the word "trial," because all I'm doing is trying the lens and then describing my results to the reader. A "test" implies a scientific approach that is experimentally sound, measurable, and repeatable. Erwin Puts names that approach in honor of Geoffrey Crawley (Editor of the British Journal of Photography for a 21-year tenure), but it could just as accurately be called the Erwin Puts Approach.

"The Crawley-Puts Approach" evaluates the technical properties of the lens in isolation, without even muddying the waters with the contributions of the imaging substrate (film or sensor), much less all the other elements of the imaging chain. The Johnston Approach, as Erwin names it (I'm flattered, even if I'm not sure he means it as flattery!), has the advantage of being more practical, and the disadvantage of being more limited. For instance, I usually include in my lens reviews disclaimers as to what they will not shed any light on—starting with color transmission, since that's largely invisible to the black-and-white films I normally use.

The much more rigorous Crawley-Puts method of describing a lens is ultimately more accurate, as well as more readily applied to differing applications, but has the drawback that it might not be descriptive of what users will actually experience using the lens for their own work. Indeed, this is often reflected in Erwin's writings when he notes that high levels of technical skill are necessary to extract the very best out of any particular lens (and sometimes to detect the differences and distinctions he describes). This is a repeated refrain in his writings, and it's obviously something that concerns him.

Armed and dangerous
The key to writing a good subjective review—the kind I write—is to arm the reader with as much information about the terms and conditions of the review as possible. My recent controversial pair of posts about the Leica M8, for instance, were widely criticized. Some criticisms were simply errors or were based on errors (one commentator concluded that the vehicle in the first picture in the reviews, of a parked BMW SUV, must be my own car. He then used this premise to speculate about my feelings toward German technology. (I drive a Ford.) Another commenter said that I work for Apple, and another stated that I write for Outdoor Photographer magazine. Neither of those things are true either). But many people made criticisms that were very valid. Even certain friends complained that I hadn't done enough shooting with the M8, for instance. I ran into repeated references on other forums saying I had shot only "90 pictures" or some number close to that. Actually what I said was that I filled up most of a single 1GB SD card, which might be 200 or 300 shots when you consider the ones I deleted as I went along—careful reading without assumptions is still required. Regardless, the underlying criticism that I hadn't shot enough with the camera still holds.

But how did they know that? They knew because I told them, that's how. I'm always amused, after I write a subjective review, how many people raise objections using the very information that I deliberately provided for them in the text of the review.

That's how a subjective review works best, in my opinion. Readers should be well informed of the writer's prejudices, the extent of his trials, the conditions under which he worked, even his tastes. They can then take all that information into account when evaluating and applying—or, yes, dismissing—his conclusions.

Called for and needed
The Crawley-Puts Approach admits of no such "slop." There is little room for impressionism. When a technical measurement is made, it must be made exactly: it's not good enough to say "Well, I wasn't being very careful, so maybe the optical resolution of the aerial image two-thirds out from the center is 10 or 20 lp/mm better or worse than my figures show." No. That won't do. Exactitude and rigor are called for, expected, and needed.

In the end, with this approach, what the reader is left with is a description only of the lens and its potential performance, and there is no guarantee that he will see this potential realized in his own work. But the description he does get is complete, thorough, and exact—more so than a subjective trial could be.

There is room for both approaches in reviewing, of course.

I've said nothing about what Erwin calls "A Third Way," but I recommend that you read his comments carefully. What he describes there is, among other things, one of the fatal downfalls of most high-end hi-fi reviews.

A final point could be raised, which is that it is very rare for any single review to be 100% of one type or the other. I, and other subjective reviewers, sometimes employ technical measurements to discover or support a point of argument. And, even in a technical review, tastes and personal judgments will sometimes be found. It is perhaps understandable when a subjective reviewer is not as careful as he should be with technical measurements, but that's no excuse. And it might also be understandable when a technical reviewer is influenced by personal tastes or previous experiences yet casts these things as facts, but he should be vigilant against that error. We all try, I'm sure.

The timing of this article is fortuitous for me, too. I'm about to embark on a new review project, and these discussions make a good foundation for that review.

Posted by: MIKE JOHNSTON