This is a hard question! I think if you are looking for a specific cutoff point for a numerical metric, it's really a Sorites paradox. It's not that there's an obvious cutoff, just a gradual shift toward more and more cruelty.

Jonathan Safran Foer compares it to the famous Supreme Court justice Potter Stewart's take on obscenity (it's hard to define, but you know it when you see it) which seems right to me:

Like pornography, factory farming is hard to define but easy to identify. In a narrow sense it is a system of industrialized and intensive agriculture in which animals — often housed by the tens or even hundreds of thousands — are genetically engineered, restricted in mobility, and fed unnatural diets (which almost always include various drugs, like antimicrobials). Globally, roughly 450 billion land animals are now factory farmed every year. (There is no tally of fish.) Ninety-nine percent of all land animals eaten or used to produce milk and eggs in the United States are factory farmed. So although there are important exceptions, to speak about eating animals today is to speak about factory farming.

More than any set of practices, factory farming is a mind-set: reduce production costs to the absolute minimum and systematically ignore or “externalize” such costs as environmental degradation, human disease, and animal suffering. For thousands of years, farmers took their cues from natural processes. Factory farming considers nature an obstacle to be overcome.

I know this might not answer your question, which I take to be: how do people choose which farms to count when they come up with a number like X number of animals live on factory farms? But I suspect that the people making those estimates don't have a clear and simple heuristic either. Which might seem pretty non-rigorous, but I don't think their conclusions are far off in terms of magnitude.

I found one source that offers a fairly simple definition. It just considers every CAFO a factory farm, which is defined by the USDA in terms of the number of animals: "at least 125,000 chickens raised for meat, 82,000 hens used for eggs, 2,500 pigs, 1,000 cows raised for meat, or 700 cows used for dairy." The # alone might not entail cruelty, but I would guess its hard to run a profitable business with that many animals without making some welfare concessions.