www. O S N E W S .com
News Features Interviews
BlogContact Editorials
.
Modern tech product reviews are flawed
By Thom Holwerda on 2017-11-08 23:09:42

Khoi Vinh on why 24 hour or even weeklong reviews are dumb:

However I've come to believe that there's at least one thing wrong with this whole notion of product reviews - and with smartphone revirews in particular - and that's that by and large they’re only ever interested in these phones when they're brand new.

When an iPhone debuts it's literally at the very peak of its powers. All the software that it runs has been optimized for that particular model, and as a result everything seems to run incredibly smoothly.

As time goes on though, as newer versions of the operating system roll out, as there are more and more demands put on the phone, it inevitably gets slower and less performant. A case in point: I'm upgrading to this iPhone X from a three-year old iPhone 6 Plus and for at least the last year, and especially over the last three months, it has struggled mightily to perform simple tasks like launching the camera, fetching email, even basic typing. People who have recently had the misfortune of having to use my phone tell me almost instantly, "Your phone sucks."

You could argue that three years is an unrealistically long time to expect a smartphone to be able to keep up with the rapidly changing - and almost exponentially increasing - demands that we as users put on these devices. Personally, I would argue the opposite, that these things should be built to last at least three years, if for no other reason than as a society we shouldn't be throwing these devices away so quickly.

This is, of course, the reason behind the odd embargo strategy Apple employed regarding the iPhone X - if you only give people an hour or at best, 24 hours, to review a device, people will still be in the honeymoon phase of owning a product, where you're still rationalising spending €1200 for a phone (or any other high price for any other product, for that matter). Choice-supportive bias is a real thing, and each and every one of us experiences it. During this period, initial flaws aren't as apparent, and long-term flaws or flaws that only pop up in specific situations aren't yet taken into account. It makes the product appear better than it really is.

This is why, back when I still did reviews for OSNews, I had my own rule of using a product for at least four weeks before publishing a review. This gave me enough time to get over this initial phase, and made sure I had a more levelheaded look at the whole thing. We don't do many reviews anymore - I have to buy everything myself, and I'm not rich - so it's not an issue at this point, but even if companies were to approach us today for reviews, I would still ask for that four week period, and if they were to object - sorry, but no review.

This is, of course, what the major publications should've done. Nobody forced The Verge or whomever else to publish a review within 24 hours. The initial embargo rush is important for the bottom-line, I get that, but it still feels rather suspicious. What can you really learn about a product in just 24 hours? Can you really declare something "the best damn product Apple ever made" after using it for less than a day? At what point does writing most of the review in advance before you even receive the product in the first place, peppering it with a few paragraphs inspired by the 24 hours, cross into utter dishonesty?

By reviewing products in a day or less, popular tech media is really doing readers and consumers a huge disservice, only further strengthening the idea that the tech press is often nothing but an extension of a company's PR department. This erodes credibility, and in turn hurts those among the media who do take their time to properly review a product.

It's okay to not rush writing a review to meet some asinine embargo. It's okay to not ask "how high?" when a company tells you to jump. It's okay to publish a review a week or even a month after an embargo has been lifted. It's okay to not post unboxing videos of non-retail boxes.

It's okay to, sometimes, just say no.

 Email a friend - Printer friendly - Related stories
.
Read Comments: 1-10 -- 11-20 -- 21-28
.
Awkward final sentence.
By woegjiub on 2017-11-08 23:57:00
You probably don't know this, but Australia has just had a nation-wide opinion poll paid for by the government regarding Marriage Equality.

And the "oppose" voice's cactchline was "It's OK to say 'No'"

Made for an odd reading of your end-point from an Australian perspective :p
Permalink - Score: 2
.
Two different points
By tanishaj on 2017-11-09 00:38:09
While I don't disagree with Thom's comments, they seem disconnected with what I took away from the article.

It is not just that reviewers only use the device for 24 hours. They could use it for 4 weeks and still be impressed.

The core point of the article, in my view, was that we should be writing product reviews of hardware that has already been in the market for a year or more. That is, after all, when a great deal of the market will first purchase the device. Reading an article about a device that was released 2 years ago will give an unrealistically positive impression versus the actual experience of owning that device today.

I am an iPhone 6s owner now. I agree with the article. Although the specs say my phone should still be decent it is, in practice, a terrible tool. It is unreliable, erratic, and performs poorly enough that it impacts my ability to use it at all in many day-to-day situations. If I wrote a review about my phone today, I could not recommend it. This is a device that was arguably the best in the market not all that long ago.

Edited 2017-11-09 00:38 UTC
Permalink - Score: 5
.
RE: Two different points
By patrix on 2017-11-09 01:09:02
Counter-point, I just recently got a 6s (to replace a 1.5-year old midrange Android which surely affects my perspective, not coming from top of te line flagsip), and I find the 6s performs about on-par as an iPhone 8 (I used an 8 for a week before going for the 6s).

Note that I don't do social media, and got that free battery replacement from Apple due to unexpected shutdown issues (which I didn't experience but previous owner did). Performance is great, typing is great, no complaints here.

(And this is why we need multiple reviews, too, because experiences vary!)
Permalink - Score: 1
.
Strange how different people's experiences can be...
By gan17 on 2017-11-09 01:49:22
> A case in point: I'm upgrading to this iPhone X from a three-year old iPhone 6 Plus and for at least the last year, and especially over the last three months, it has struggled mightily to perform simple tasks like launching the camera, fetching email, even basic typing.

I'm also on a three-year old iPhone 6 (non Plus though) and my experience has been better than the authors. Heck, I'd say the 6 performed worst with iOS 8 (not a version iOS historians look back on fondly) that it shipped with. It improved considerably with iOS 9, and stayed on par throughout iOS 10. Now currently on iOS 11.1, and aside from the occasional stutter when typing (something that's been there with every iOS version this phone has had), some apps taking a bit longer to launch, apps being forced to reload more often when multitasking (only 1GB RAM) and battery life that's representative of the degradation from three years worth of charge cycles, it's been pretty decent in daily use. I must be doing something wrong.

Other than that bit, I agree that most of these "early" reviews are dumb. Lets not even talk about that iMore reviewer, shall we. Apple could released bottled water tomorrow and the guy would say it's the best water ever made, tasting more watery than any water that's come before.

Edited 2017-11-09 01:52 UTC
Permalink - Score: 4
.
Making money
By mdsama on 2017-11-09 04:30:08
I completely agree from a quality-content point of view, but commercial realities for publications are complicated...

There are probably principled publications that refused the 24-hour review, but they aren't noticed for it. (They just have no review.)

And these short, all-hype reviews are probably more widely read (generating more ad revenue) than in-depth articles. As long as there's money to be made in publishing that sort of content, it's probably going to continue.

It's annoying, and it's worth complaining about (because backlash is one of the few counterbalances to how successful cheap content is)... I guess I don't disagree, but also find it hard to blame the publications
Permalink - Score: 3
.
Agree mostly
By Bill Shooter of Bul on 2017-11-09 05:33:06
I think another aspect missed in Thom's listing, is that reviewers are experts in things. They care way too much about things most people simply do not.

Most people when looking at an image couldn't tell you which phone it came from, and wouldn't care if it were more saturated, or pixilated or any of the other photo enthusiast terms.

They also don't care about ui consistency, as long as its obvious how to do a specific task.

Yeah Facebook is a different UI than Twitter. And Apples have to be eaten differently than oranges, And planes have different controls than bicycles. Sometimes different is better, or at least not impossible or annoying to most people.
Permalink - Score: 3
.
Yes sir
By Poseidon on 2017-11-09 07:06:38
I agree with the assessment. I’d take the iPhone 8 any day over the X. Same SOC with Touch ID.

To properly asses a product you’d need at least a month on all use cases and with daily and weekly observations.
Permalink - Score: 3
.
Dunno
By stereotype on 2017-11-09 09:50:27
First world problems?
Who cares?
It amazes me how we just nod and play the games these companies want us to...
Permalink - Score: 1
.
RE: Two different points
By Savior on 2017-11-09 10:04:36
> While I don't disagree with Thom's comments, they seem disconnected with what I took away from the article.

It is not just that reviewers only use the device for 24 hours. They could use it for 4 weeks and still be impressed.

The core point of the article, in my view, was that we should be writing product reviews of hardware that has already been in the market for a year or more.


I didn't read the article itself, but just from the lead quoted here, that is also what I thought the article was about. And it is a perfectly valid point, although I don't see how year-long review periods would be feasible in today's "I want it now" Zeitgeist. This might especially be true for Apple-fanbois.

Such reviews are not completely unheard of, however. At least with cars, one can find reviews that start with "I have had this car for X years...". Unfortunately, it is not the norm with other appliances.

Edited 2017-11-09 10:05 UTC
Permalink - Score: 3
.
Makes me think of old audiophile reviews
By number9 on 2017-11-09 11:55:51
This reminds me of the post-golden hifi review days. Several of those magazines, love them or hate them, would review equipment for at least a full month (this was the bare minimum) before writing a review. Several of them would describe the months they spent reviewing and comparing the item under review to their reference systems and other gear.

Of course, the pace of audio innovation was much slower than computers and phones are today.
Permalink - Score: 1

Read Comments 1-10 -- 11-20 -- 21-28

No new comments are allowed for stories older than 10 days.
This story is now archived.

.
News Features Interviews
BlogContact Editorials
.
WAP site - RSS feed
© OSNews LLC 1997-2007. All Rights Reserved.
The readers' comments are owned and a responsibility of whoever posted them.
Prefer the desktop version of OSNews?