There's a small philosophical literature on the ethics of sweatshops. There are some libertarian type people who defend basically the view you describe, that if people voluntarily work at these places, they must represent their best option, so sweatshops can't be morally wrong. This paper is representative of that view: https://philpapers.org/rec/ZWOSCA-2

Then there are people who try to argue that sweatshop labor really is wrong nonetheless. One idea is basically Kantian -- sweatshop owners treat their employees as a mere means to an end, which is wrong. Another idea is that sweatshop workers are wrongfully exploited. For instance, lots of people have the intuition that in a voluntary exchange, one party shouldn't use their bargaining power to reap all the gains from the trade. I.e. they think there is a "fair price" where buyer and seller both get an equitable share of the benefits.

Just for illustration, imagine I have an old Camaro that I'm willing to sell for $5,000. You're willing to buy it for up to $100,000. If I know you desperately need it, and I negotiate you up to $99,999, then I'm getting almost all the benefits from the trade (you are one dollar away from being indifferent about the exchange). Lots of people think that's wrong. This might be the intuition behind thinking price gouging is wrong, too.

So you can argue that sweatshops are basically similar (although that's an interesting empirical question). Maybe sweatshop laborers are just barely better off than their next best option, while the owners are profiting a huge amount from their labor. People might think that's exploitative, and that for things to be equitable both parties need to get a reasonable cut of the benefits from the exchange. This paper has some more discussion you might find interesting: https://philpapers.org/rec/MAYSEA

I slightly cringe when people lead with the “sweatshops are great” thing. I mostly* agree with the point, but it can seem like a combination of

1. Self serving motivation
2. Trying to “own the libs”/make a rhetorical point rather than doing the most good

*I also have some sympathy for the fair trade-ish argument that consumers could use their bargaining power/government negotiating trade deals … that could give lead to better pay for the poorest workers in the world.

So yeah, better to buy from sweatshops in India than buying American… but maybe pressure from the anti sweatshop movement could induce big companies to pay the Indian workers a little bit more. (At least it seems possible, I’m not saying it will necessarily work.)

There could be issues of informed consent in some cases, because workers may not be aware of the health risks posed by particular sweatshop work. Some sweatshops may use forced labour. In both cases, workers may not have come to a well-informed judgement that it's the best among available options, or they may come to decide against it but can't leave (because of forced labour).

I don't know how common these are or how much they figure into the popular opposition to sweatshops, though. I think it's more the impression of exploitation and generally harsh working conditions, especially with child labour, even if workers take the work on voluntarily for good reason.

Whether sweatshops are good and whether we should buy from them are two different questions. I think it's perfectly consistent to say sweatshops are terrible, but boycotting sweatshops only makes things work. Like some of the other comments have said, sweatshops are exploitative, sweatshop workers are dehumanized, and informed consent is a serious issue, to put it mildly (I've read about young girls essentially forced by their families into multi-year contracts that they cannot escape from, in order to pay off family debts). People who reject the idea that sweatshops are "good" or even a "net positive" aren't totally wrong -- in a genuinely good society there should not be any sweatshops. 

That said, buying from a local company that makes you feel good does not do anything to help the sweatshop workers. At the end of the day, saying we should boycott sweatshops is saying we should put the workers out of business, and saying we should buy locally means we should keep our wealth within our wealthy communities.  Buying locally is a self-centered solution because it focuses on alleviating the feeling of guilt at benefitting from exploitation of the poor.  But it's probably not the right response for someone genuinely seeking to benefit those poor.  

Sweatshops are probably a "net positive" compared to buying locally, but they are not a "net positive" compared to factories that treat their workers well and share the profits fairly. People may be put off by the framing of sweat shops as "good" because it sounds like saying that the exploitation of poor workers is desirable. A better framing might be that the exploitation is already happening, and buying from sweatshops is likely better than boycotting if you're concerned about the welfare of those workers. But that's not to say that sweatshops are not sites of many terrible abuses, and that seeking to eliminate those abuses through other avenues would not be a good thing. 

You may not be particularly interested in moral judgments of sweatshops as good or bad if you are focused on the utilitarian question of whether they are causing a net harm or a net good, but acknowledging that they are "bad" in the sense that they are exploitative and abusive may make it easier for people to swallow the separate argument that buying from sweatshops may nonetheless be more beneficial than harmful to the workers.

Probably because those who say sweatshops are great tend to not work at sweatshops. 

So it seems obvious to others that there are better ways for people to live, yet not everyone has access to those options. And arguments in favor of sweatshops are seen as missing the point or even keeping it that way.