When split-testing a listing element (most often the main image), the comparison set should include competitor images, not just the seller's own current/prior version. A variant that beats the seller's own old image can still be non-competitive on the search results page if it's weaker than what competitors are already showing — testing only against oneself risks mistaking a local improvement for real competitiveness. This applies whether the test runs via off-platform panel tools (which can mock up realistic search-result-page layouts for this purpose) or Amazon Manager Experiments.
For sellers unsure where to start, the recommended entry point is: audit a live listing for general qualitative feedback, then test the main image head-to-head against the top three competitors — not just against your own prior image.
The full iterative loop is four steps: (1) test your current main image against a competitor's image; (2) if you lose, generate several new image variations addressing the loss; (3) test the variations against each other to find an internal winner; (4) retest that internal winner against the same competitor to confirm it actually beats the competition, not just your old image. Skipping step 4 risks a false 'local maximum': "if you don't test it against your competition, you might not have made any improvement, right? It just looks better relative to what it was before." See Helium 10 Audience Tool (Shopper Image-Preference Survey) for the platform this is typically run on, and Main-Image Tactic Checklist for what to vary between iterations.
Recommended second step after the listing audit: pit the current main image directly against the main images of the top three competitors using a neutral "which would you buy" question with light category/keyword context — no new creative needed. The value lies less in the win/loss count than in the stated reasons shoppers give for preferring a competitor's image.
Reinforces why self-only comparison is insufficient: testing only against your own past creative can produce a false sense of improvement (a local maximum) that still loses to competitors.