Sure, we could use some fancy software to spit out all kinds of data about the cameras we test—but a bunch of numbers don't always mean so much when it comes to actually using the camera. When we test out a new camera, we put it through a few real world tests along with a few more rigorous standardized tests to give you the ins and outs of each camera. Here's how we test for our hands-on reviews.
How We Test: Real World Testing
Chances are, you aren't looking for a camera to take pictures inside some carefully controlled laboratory setting. We take each of our hands-on cameras and put them to use in as many everyday scenarios as we can. We shoot pictures indoors and outdoors. We try some macro shots and landscapes. We try portraits, still life and action.
As we shoot, we ask how the camera handles that particular environment. When we shoot outdoors, we're asking things like: Can you still see the viewfinder or LCD screen in bright sunlight? Does the camera handle bright light mixed with darker shadows well? But most cameras can shoot pretty decent images in the day, so we also try more challenging scenarios, like low light indoor shots. Can the camera capture decent indoor shots without a lot of noise? Does the autofocus slow down in low light? Does the flash light up something that's all the way across the room?
We go through as many of the camera's features and settings as we can in the time that we have with the camera. (We get each of our review models on loan directly from the manufacturer and can spend anywhere from two weeks to two months with one camera). If a camera has manual modes, we shoot with the manual modes as well as testing out how well the automatic settings perform too.
Along with looking at how the camera performs, we also pay attention to how easy it is to use. Is there a quick menu to adjust settings easily? Is the layout of the controls confusing?
How We Test: Image Quality
After our real world testing, we take a look at our images and see just how well the camera performed. Are the colors accurate? Are the images in focus? Do the photos look better or worse than we expected?
Along with our real world tests, we also use a few standardized tests to make cameras easier to compare. One of those tests evaluates sharpness. A camera with excellent image quality can take a picture of a test chart with lots of lines and look nearly identical to the actual, printed test chart. The lines will be sharp and you'll be able to see separate lines. A camera with poor image quality, however, will make it difficult to distinguish between fine lines.
Click on these images to view them at full size:
Both images were shot at a similar distance from the chart and cropped in a similar manner—no edits were made to enhance the sharpness in any way. In cameras with manual modes, we set the aperture to f/8 so that the depth of field does not negatively impact the sharpness. Notice how, on the image from the iPhone, the top horizontal line on the left looks like one line. It's not. If you look at the image from the Nikon D3300, you can clearly see separate lines there that get closer together as they approach the center. In the image sharpness test, we would rate the iPhone at poor and the D3300 as excellent.
There is more to image quality than just sharpness, however. We also test a camera's white balance by shooting a white balance card with all of the camera's different white balance settings. If the camera has a manual preset, we compare that to the image of the automatic white balance setting to see how accurate it is. In the image, the proper white balance will have the white space on the upper left of the card as a pure white.
Another factor in image quality is noise, or that grainy look some images can have. Noise is greater at higher ISOs. We take images at each ISO level and determine where the noise begins to be distracting. Some cameras can shoot at ISO 1600 without overwhelming noise while others should stay under ISO 800 as much as possible.
How We Test: Speed
Performance is a big factor in rating a camera. While we test a camera's performance in our real world tests, we also put the review cameras up to a few speed tests.
Shutter lag is the amount of time it takes to take another picture. To test this, we take several, single images in full resolution as fast as we can of a stopwatch. We average the times together to get an accurate idea of shutter lag. Cameras like point-and-shoots we expect to have a bit longer time between shots, while DSLRs tend to be faster.
Speed is also important when it comes to autofocus. To test autofocus speed, we set up a test similar to our shutter lag test—the only difference is that we take a picture of a stopwatch and a second object and refocus between the two items in between shots. The result of this test minus the shutter lag speed gives us a good indication of how much time, realistically, it takes to focus and snap the picture.
A camera's processor slows the more pictures you take in a row. We test the burst speed of a camera both to count how many images you can take in a row and to see how soon you can take more images. For example, the Sony HX400 can take 10 pictures per second, but takes about 12 seconds to process all those pictures before you can take more. If you use the slower burst option, it still takes ten pictures in succession, but only takes about nine seconds after processing to snap another one.
How We Test: Choosing a final grade
All of our hands-on review cameras receive a letter grade. Here's the four categories we take into consideration when rating a camera:
Body and Design: How well is the camera designed? Are the extra features worth the added weight and size? Is the control layout easy to use? Is the camera larger or more compact than other similar cameras?
User Experience: How easy is it to use the camera? Are there any controls, features or settings that are frustrating to use? How good is the battery life? Overall, is the camera enjoyable to use? Does the camera have enough features to please an enthusiast or is it more for an average consumer? Can the camera take a variety of different types of shots? How fast is the camera?
Image Quality: How well did the camera fare in the image sharpness tests? Does the camera perform well at high ISOs? Can the camera take good images in low light? Does the camera reproduce colors accurately? Do the images have good detail?
Value for Money: How does this camera compare to other similarly priced cameras? Are there any features that are missing that we would expect to see on a camera for this price?
The answers to these questions will be listed quickly in the Pros and Cons section, as well as fully detailed within the review. It's important to check out this section before buying, because choosing the right camera is sort of like dating—you have to find the one that suits your personality. If you shoot a lot of sports and kids, for example, stay away from cameras that list speed in the cons section, even if it's rated well.
When we rate cameras, we take into consideration the price and category. We may give a point-and-shoot an A- and a DSLR a B+--the DSLR will get you better images than the point-and-shoot, but it's also pricier and has more competition.
Here's a break-down of what our grades mean:
A+ Wow. Just Wow! A Excellent camera for the price.
A- Very good, with a few minor flaws.
B+ Good, but there's competing models with better specifications.
B A solid camera, but certainly not the best in the category.
B- An okay camera, but if you can afford to, look for a better one.
C Nothing too special about this camera.
Under a C There's a big flaw with this camera, or it's very old and not worth a look compared to what's on the market now.
We don't sell cameras, so we don't make more money if you choose to buy one camera over the other. We receive the funding to continue running our site through link clicks—but it doesn't matter to us if you click on a link for a $100 camera instead of a $500 one. We never receive gifts or payment for writing reviews. Our goal is to match you up with the right camera for your needs through honest, unbiased reviews.
Got questions? We'd love to chat. Leave us a comment below.