Jared Spool on how many users you should test with.

The other day, I saw in a lecture that was given at Harvard, that you can find 80% of the problems if you test with 8 users or fewer. That was never a true statement. It was closest back in 1992 when Jim Louis and Bob Versey first did it, but that was based on 1992 systems. We were looking at applications that might have — if you had 10,000 users, you are one of the biggest applications in the world. You are one of the top 10 applications that had ever been sold. And now, we have websites that tell us that they have billions of active users every day and you’re going to tell me that eight people are going to find 80% of the problems?

So, if we just scale from 10,000 users to a billion which is a — six orders of magnitude, then, our eight users should now be six orders of magnitude bigger. We’re now looking — I don’t know what’s eight to the six.

You could — you could get me to buy into the idea you need a minimum of 262,000 users to get to 80% of the problems.

I might buy into that.

— Jared Spool