Richard,
Different libertarians have different views about utilitarian calculation. Many libertarians, especially those who focus a great deal of attention of economics, are in fact philosophical utilitarians, and make utilitarian arguments for libertarianism. I happen to think that they are wrong about that but they are certainly out there.
Libertarians who are opposed to utilitarian calculation, and who criticize it as “collectivist,” may be criticizing any of several related but distinct things, since “collectivism” is a cluster concept that has several related but distinct applications. Classical utilitarianism is individualist in some senses and collectivist in others.
Speaking for myself, I think there are at least two ways in which classical utilitarianism can be described as objectionably “collectivist.”
(i) The first is indeed related to the standard sacrifice cases that you discuss here. But I think your statement of the case in terms of balancing “interests” obscures the real issue. Libertarians who object to utilitarian calculation aren’t entering the debate over whether or not Bob’s “interests” can ever be jointly outweighed by Alice’s, Ted’s, Norah’s, etc. in moral deliberations. It has to do with whether certain kinds of “interests” can ever outweigh certain other kinds of “interests.” In particular, whether the benefit that Alice, Ted, Norah, etc. can get by forcibly seizing some natural good that rightfully belongs to Bob, can outweigh the violation of Bob’s individual liberty of person and property. (Or, more tendentiously, the violation of Bob’s right to control his own labor and enjoy the fruits of it.) Principled individualist libertarians (1) deny that the one kind of benefit outweighs the other[*], and (2) argue that the the reason has to do with qualitative distinction between the kinds of benefit and harm, not just a quantitative difference in the amount or the reliability or whatever of utility.
[*] I’ve left it open as to whether the first condition is supposed to be a universal statement or a statement of typical conditions, because different libertarians differ on this point, depending on how absolutist they are and on their views on emergency situations.
That’s very far from being a crazy position to take, even for a committed utilitarian. Lots of utilitarians have wanted to make qualitative and not merely quantitative distinctions between different kinds of benefit and harm. As well they should. (Here’s a case: suppose you were deliberating whether to rape a child on live television, and that the broadcast would only be played for convicted pedophiles. Presumably the victim would endure suffering from the rape and the audience would get some small pornographic thrill from watching. If all you can do here is run a simple quantitative comparison of total suffering to total enjoyment, then whether or not you should rape the child becomes a matter of whether or not the audience is large enough that the sum of all the small thrills adds up to more than the total suffering inflicted on the victim. And for any non-zero level of thrill that audience members might get, there must be some hypothetical audience which would be large enough to justify raping the child.) I take it that this sort of worry is why many utilitarians have been inclined towards making at least some qualitative distinctions between different kinds of benefits and harms. Also why many of the attempts that have been made (e.g. weighting the calculation so as to take the distribution and not just the net global total of utility into account) have tended to try to bake in some limitations on how badly any individual person can get treated for the profit of other individual people.
Of course, you might agree with the principle of having some limitations on how individuals can be treated in order to maximize total net utility, but reject the claim that the right limitations to have line up with the limitations suggested by libertarian rights theory. Libertarians will argue that other baselines for treatment of individuals don’t guarantee enough respect for their individuality, or the integrity of each person’s one and only life. That’s a substantive argument to be had elsewhere, but I think that the libertarian position here is certainly intelligible.
(ii) The second doesn’t actually have anything directly to do with sacrifice cases, although I think it explains why many people think that certain sorts of sacrifice cases are morally acceptable, when in fact they are not. To wit, utilitarianism methodologically begins with the idea that the object of ethical deliberation is maximization of global good. I think this is mistaken: the object of ethical deliberation is something much more personal, viz. the best way to live. So my worry here is not whether or not utilitarianism is “collectivist” in its treatment of moral patients, but rather whether or not it is individualist in its treatment of moral agents.
Thus, for example, utilitarians (including both non-libertarians and some utilitarian libertarians) claim that there are cases where you can justifiably coerce Bob in order to stop a greater or a worse degree of coercion against Alice, Ted, Norah, etc. I think this is a fundamental mistake: the first task you should set yourself to is not keeping down the total amount of injustice going around, but rather not being unjust. Swallowing the personal demands of ethics up into some global mission to maximize net good ends up doing violence to the individuality of the moral agent. (As you may have noticed, it’s won’t take a very long segue to get from here to Bernard Williams’s objections against utilitarianism. So how you feel about those may determine how you feel about the concern here. But whether it’s right or it’s wrong, again, I hardly think it’s a stupid or unintelligible position.)