Imagine being asked to give your opinion on a book after reading only the first sentence, or to rate a movie based solely on the opening credits, or to describe Van Gogh’s “Starry Night” after only a glimpse at the painting’s top left corner. In effect, this is what you ask test participants to do when you use an image that requires scrolling (horizontal or vertical).
In any situation involving images, a person can’t comment meaningfully on your design if (s)he can’t see it in its entirety. The more you require the participant to scroll an image, the greater the likelihood of getting non-responses to your test questions.
While image scrolling has a negative impact in all tests formats, the greatest effect is in target identification tests and in mixed format tests that include target identification questions. In most instances (or unless the test instructions indicate otherwise, participants are inclined to internalize specific elements of a design first. The process of scrolling an image takes focus away from the image and places it on the actions required to scroll.
Researchers always have to be thoughtful about how the image is presented within the test. When testing in-person, it’s a simple matter of knowing what monitor will display the test image and customizing the image to fit the screen resolution. When using a unmoderated testing tool (like this one), you’ll need to use an image that is likely to not require scrolling when used in the most frequent screen resolutions (doing an internet search on “browser display statistics” should provide a number of current resources to reference as you consider the technologies of who might be taking your tests).
In order to make it fit, you’ll most likely need to do some image resizing. Cropping the image will preserve the detail necessary to provide a realistic view of the design, but provides only a segment of the image at the cost of losing the context of the image as a whole. Scaling will retain the entirety of the image, but could degrade the image detail so much that the image becomes too distracting to elicit meaningful feedback. (If one or the other does not suffice, combining the techniques may provide an acceptable balance of preserving context and presenting discernible detail.)
As always, the main concern is to make sure that the participant is put in the best position possible to meaningful feedback about the design.