Act utilitarianism states that the best act is whichever act would yield the most happiness. Rule utilitarianism instead states that the best act is to follow the general rule which would yield the most happiness.
To illustrate, consider the following scenario: A surgeon has six patients: one needs a liver, one needs a pancreas, one needs a gall bladder, and two need kidneys. The sixth just came in to have his appendix removed. Should the surgeon kill the sixth man and pass his organs around to the others? Or, indeed, what would stop him from simply hunting down and slaughtering the first healthy man (the seventh) he comes across on the street, patient or non-patient? This would obviously violate the rights of the sixth/seventh man, but act utilitarianism seems to imply that, given a purely binary choice between (1) killing the man and distributing his organs or (2) not doing so and the other five dying, violating his rights is exactly what we ought to do.
A rule utilitarian, however, would look at the rule, rather than the act, that would be instituted by cutting up the sixth man. The rule in this case would be: "whenever a surgeon could kill one relatively healthy person in order to transplant his organs to more than one other person who needs them, he ought to do so." This rule, if instituted in society, would obviously lead to bad consequences. Relatively healthy people would stop going to the hospital, we'd end up performing many risky transplant operations, etc., etc. So a rule utilitarian would say we should implement the opposite rule: don't harvest healthy people's organs to give them to sick people. If the surgeon killed the sixth (seventh) man, then he would be doing the wrong thing.
In every system of morality, which I have hitherto met with, I have always remark'd, that the author proceeds for some time in the ordinary ways of reasoning, and establishes the being of a God, or makes observations concerning human affairs; when of a sudden I am surpriz'd to find, that instead of the usual copulations of propositions, is, and is not, I meet with no proposition that is not connected with an ought, or an ought not. This change is imperceptible; but is however, of the last consequence. For as this ought, or ought not, expresses some new relation or affirmation, 'tis necessary that it shou'd be observ'd and explain'd; and at the same time that a reason should be given; for what seems altogether inconceivable, how this new relation can be a deduction from others, which are entirely different from it.