Stain Scores and Streaks: How We Developed a Better Mopping Test

We’ve been a bit obsessed lately with finding a reliable way to objectively test and score robot vacuum mop combos and upright electric mops. While we do evaluate how well these machines tackle stains, we’ve also realized the need to measure water usage—since using too much water may lead to streaks on your floors or possibly leave messes behind. So in this review, we are taking this challenge head on; explaining why water usage matters, walking you through our new testing methods, and revealing how these findings have changed our rankings and overall methodology!

Tackling Stain Tests

In the last year or so, we have been working hard to find a way to objectively test and score robot vacuum mop combos and upright electric mops. We developed a system to do this a while back, where we set up nine stains—four coffee and five grape juice stains—and carefully measured out increasing amounts of fluid for each stain using a syringe.

Each stain is assigned a points value based on its level of difficulty, and each robot is given four attempts to remove the stains: the points earned on the first run are multiplied by four, on the second run by three, on the third run by two, and the points on the final run are added at face value. There was also a modified version of this test for upright electric mops, which we explained in a recent review.

We have tested a ton of products this way over the last few months, and the results of those tests have dramatically impacted the rankings of the robot vacuums on our Top 20 Robot Vacuums list and our Top Hard-Floor rankings as well.

These results even led us to declare that, since robot vacuums are as good—if not better—than these upright hard-floor cleaners (which the various tests we did seemed to indicate), it seemed like a no-brainer to buy a robot vacuum mop over a hard-floor cleaner mop in most situations because robots are automated and require less maintenance. However…

The Major Flaw: Water Usage

There was a major flaw in that thinking that we should have caught early on, and that flaw is water usage.

Basically, the more water that a product puts out when mopping, the better it will do with these dried-on stain tests. But that’s a problem because more water left behind means more streaks on the floor, and that essentially means the floors are left more dirty if the products use a ton of water.

This creates a perverse incentive for companies to program robots to use a lot of water when mopping in order to do well on these kinds of tests. They don’t have to think about the water left behind as much, since tracking streaks and water usage isn’t something that anyone is doing as far as we know.

This is a bigger problem for robot vacuums that use spinning pads or fixed pads, because they don’t deal with lots of liquid well—they just kind of spread liquid around if there’s too much of it. So it’s critical for those types especially to use precise amounts of water to mop the stains up, but not leave much water behind. It’s a balancing act.

With roller mops, however—like on upright hard-floor cleaners or robot vacuums that have rollers—it’s a bit different. They can use a lot of water to clean the stains, but since those systems squeegee out excess water from the roller as it mops and store dirty water onboard, they don’t tend to leave behind as much water.

Why It Matters

To recap, the problem here is that there are a lot of robot vacuums that Vacuum Wars has ranked really high, which should be ranking a lot lower, because the mop test we did previously unintentionally rewarded high water usage where it should have penalized high water usage.

Scoring Water Usage Objectively

So how are we fixing this problem? The first thing we did was develop a way to objectively score their water usage. We did this by weighing a paper towel before and after a run on a 4×4 section of the test area on the low and max water settings. What we found was disappointing.

Surprising Results

We discovered that robot vacuum manufacturers were all over the board with water usage. Of the thirty-one robot vacuums we tested, the average weight of the water left behind was 1 gram. However, some—like the Sharks—went as high as 3 grams, while on the lower end, like with the Samsung and the Roborock S8 MaxV, almost no water was left behind.

Water Settings Options

One really frustrating thing was that there weren’t a lot of options between their low and max water settings. If you had one that put out too much water, you were just kind of stuck with it—like with our previous number one pick, the Roborock Qrevo, where it put out 2.1 grams of water on its max setting and 2 grams of water on its low setting. So your choices were basically the same, with “too much water” or “too much water.”

We found that problem to be common, too. With few exceptions, there wasn’t that much difference between many robot vacuums’ low and max water settings.

Another interesting thing was that, although we found very consistent scores for each model (meaning that if we tested the Dreame L40 multiple times, we kept getting an average of 1.8 g of water), it was not consistent across an entire brand. Each Roborock or Dreame model seemed to have been programmed differently in this regard. For example, the Dreame X40 used consistently a lot less water than the L40. This lack of consistency across brands was disheartening. It was like each model had a new programmed water setting.

We should say that this problem is something we are pretty confident manufacturers can fix in firmware updates if they wanted to, since we have had companies tell us this directly. And we think that we should ask them to do that if they can. If we at least had the option to have a low water setting that didn’t gush water, it would fix this issue in many cases.

We should also say that this doesn’t mean that less water is always good. In fact, some of the ones that used the least amount of water were the worst at mopping, as you would expect, because you need a certain amount of water to mop stains on floors—just not too much. The main thing we are trying to say here is that manufacturers need to start trying to find this balance and be more consistent.

“If you have a robot vacuum that uses too much water, we would suggest keeping it on its lowest power setting and emailing the company a link to this video, asking them to add a firmware update to make the low water setting lower.”

Vacuum Wars

Rollers vs. Pads

This also means that robot vacuums that use rollers instead of pads tended to do better in our tests, but not necessarily. For example, the Eureka J20 and the Eufy S1—both of which use rollers instead of pads—were number one and number two respectively if you only consider the two data points of water usage and dried stain test scores.

But on the other hand, the 3i S10, which also uses a roller system, ranked near the bottom in 26th place using the exact same metrics. So while having a roller system doesn’t guarantee better results here, but it does seem to help.

Adjusting Our Rankings and Methodology

So what are we doing to fix this?

• We took down that video about the hard-floor cleaners where we claimed they weren’t as good as robot vacuum mops. They clearly are a little better if for no other reason than they’re more consistent with the small amount of water they leave behind while achieving similar stain test scores.
  
• We are now doing the following to determine how good robot vacuum mops are at mopping:
  
  1. We first run a test on its maximum water setting and get a score for both its stain removal results and its water score (in which the more water left behind, the more of a penalty it gets).
     
  2. Next, we have another tester rerun the test, this time on medium power, and score both the stain results and the water score.
     
  3. And lastly, we do an additional water penalty test for low power.
     
     and then we average these scores.

So theoretically, the mop rankings going forward should be based on the best stain removal scores with the least amount of water left behind. We have updated the algorithm, and it radically changed the rankings of the Vacuum Wars top 20 robot vacuum list, with the biggest change being that the Roborock Qrevo Curv, which was ranking number one for several weeks, has now dropped to number five. Meanwhile, the Qrevo Edge has taken its place at #1.

New #1 – Roborock Qrevo Edge

Final Thoughts

If you have a robot vacuum that uses too much water, we would suggest keeping it on its lowest power setting and emailing the company a link to this video, asking them to add a firmware update to make the low water setting lower. We will try to do much better in the future because it’s our goal to try to find ways to genuinely help consumers make more informed decisions.