This is a hard question! I think if you are looking for a specific cutoff point for a numerical metric, it's really a Sorites paradox. It's not that there's an obvious cutoff, just a gradual shift toward more and more cruelty.
Jonathan Safran Foer compares it to the famous Supreme Court justice Potter Stewart's take on obscenity (it's hard to define, but you know it when you see it) which seems right to me:
Like pornography, factory farming is hard to define but easy to identify. In a narrow sense it is a system of industrialized and intensive agriculture in which animals — often housed by the tens or even hundreds of thousands — are genetically engineered, restricted in mobility, and fed unnatural diets (which almost always include various drugs, like antimicrobials). Globally, roughly 450 billion land animals are now factory farmed every year. (There is no tally of fish.) Ninety-nine percent of all land animals eaten or used to produce milk and eggs in the United States are factory farmed. So although there are important exceptions, to speak about eating animals today is to speak about factory farming.
More than any set of practices, factory farming is a mind-set: reduce production costs to the absolute minimum and systematically ignore or “externalize” such costs as environmental degradation, human disease, and animal suffering. For thousands of years, farmers took their cues from natural processes. Factory farming considers nature an obstacle to be overcome.
I know this might not answer your question, which I take to be: how do people choose which farms to count when they come up with a number like X number of animals live on factory farms? But I suspect that the people making those estimates don't have a clear and simple heuristic either. Which might seem pretty non-rigorous, but I don't think their conclusions are far off in terms of magnitude.
I found one source that offers a fairly simple definition. It just considers every CAFO a factory farm, which is defined by the USDA in terms of the number of animals: "at least 125,000 chickens raised for meat, 82,000 hens used for eggs, 2,500 pigs, 1,000 cows raised for meat, or 700 cows used for dairy." The # alone might not entail cruelty, but I would guess its hard to run a profitable business with that many animals without making some welfare concessions.
Are you sure? My guess is that it wouldn't change much, that most factory farms are "obviously" factory farms, because there isn't a lot of economic or moral incentive to sit on the borderline.
Maybe not for most people reading the people reading the EA forum. I think if you take a serious look at the issues of animal suffering and farmed animal conditions, you'll probably arrive at a number similar to existing statistics on numbers of factory farmed animals.
But I think there's plenty of people who have motivated reasoning to doubt those statistics, or minimise the badness/factory-ness of a farm, or farming practice. For example, my extended family run a dairy farm. I remember when first reading about factory farms thinking 'well, the family farm isn't like these factory farms... right? '
I also think it's possible animal agriculturists will seize on uncertainty around the term 'Factory Farm' to sow confusion and whitewash animal welfare issues. Suppose that in the future, the concept of 'Factory Farms' gains widespread public vilification, in the same way that 'Fossil Fuels' does now. Now imagine a pan-European animal agriculture lobby group seizes on the looseness of the term 'Factory Farm' to ensure European farms aren't associated with it:
European farms aren't Factory Farms! We have better animal welfare standards here. There are cage-free policies here! Animal welfare laws! Standards and checks! It's only farms outside of Europe that are factory farms, those are the ones that should be counted in the statistics, not European farms!
I don't see this as "economic or moral incentive to sit on the borderline" but rather 'if forced to adhere to higher welfare standards, there's an incentive to maximise the reputational gain from this'.
edit: added last paragraph