The exclusionary rule is one that says if the police violate the 4th amendment of the Constitution (against illegal searches and seizures), then the remedy is the exclusion of that evidence. Sounds simple, and drastic, enough.
Practically speaking, it almost never happens. The exclusionary rule has been one which the police have used as their basis to fabricate probable cause over the last 50 years, not one which has overly restricted them in most areas they fight the most crime (note that the exclusionary rule has probably had great effect in wealthier neighborhoods where there is less crime - the police there are more afraid of the power of the residents to wreak havoc on them if they violate citizen rights there - there is little concern getting in trouble for violating rights in poor minority neighborhoods unless it's caught on camera). It used to be that the police could invade your home with impunity, arrest you, search you, violate all of your rights, and there was nothing you could do about it. In a case known as Mapp v. Ohio, the Earl Warren Supreme Court extended the exclusionary rule to all state actors (meaning all branches of state and local law enforcement).
The facts of Mapp were instructive, and they are actually recited in this new NY Times article that has spurred me to write here. The police basically went into the home of a woman they did not like, manhandled and abused her, searched it carefully, and arrested her for, get this - obscenity. If you ask me, the only obscenity was the police conduct. The Supreme Court agreed, and determined that the only way it could ever deter police conduct like this was to let them know that all of the fruits of their efforts would be in vain in the future if they did this, and hence, the said that any fruit of the poisonous tree must be discarded and cannot be used against a defendant.
One thing the Times article mentions is that this is an area in which the US gives far greater protection than in any other Western society to the accused. Evidently, the fact that we are out of sync with other countries bothers conservatives on this one (they show no similar concern when it involves things like, say, the death penalty).
However, a few things must be noted.
Police in this country are probably more politically powerful and involved than in any other country. Police unions give huge sums to politicians, police chiefs are mainstays of any politician's campaign, and few people ever dare to cross the police in any election here. What does that mean - it means that in our adversarial system, the police and prosecution hold nearly all of the cards - if the police violate the law with impunity, and they are not punished immediately, they really have little fear of lawsuits down the road, as most of these defendants make terrible plaintiffs. No one will ever take their case, and they'll probably lose anyways.
In this country, the punishments for even minor crimes can be so dramatically more severe than in other countries, the effect of even a little bit of police fudging can have the immediate effect of people spending the rest of their lives in prison. The Canadian case contrasted in the article talks about a man who was pulled over without a front license plate and the police found 77 pounds of coke in his car - he got 5 years. I can tell you, in many states, he'd get life. In California, he'd probably get 15-20 years, and in the federal system, he'd probably do more. The police mistake in that case was that the suspect's car did not need a front license plate because he was from a province that did not require front plates, but was driving in one that did so require them.
In the US, there is an exception called "good faith" (or the Leon exception, named after the Leon case). This exception has nearly swallowed the whole rule, because in a case such as the Canadian one, the officer could simply say "I made an honest mistake, I never knew that Alberta law did not require front plates, and he was driving in Ontario, so I thought he was violating Ontario law." In Canada, even a small mistake like this was a big deal, in the US, it would barely merit mention unless it was a huge crime, and even then it would only be a minor footnote. There would be no backlash against the police for that.
The final thing is, Judges are extremely reluctant to throw evidence out and anger the police, either by calling their judgment into question, or by calling them a liar. I have won only a handful of motions to suppress in my career, despite doing this for nearly 15 years, and running hundreds of them. Consider the OJ case. If ever there was a situation where the officers clearly lied as a basis to jump his wall and search his residence, this was it. They fabricated probable cause so Mark Fuhrman could then find the bloody glove on the path. The Judge in the case found that the detectives showed reckless disregard for the truth when they recounted why they did the search. Everyone clearly knew that they were lying (they also broke into his car and found evidence there). However, the evidence was not suppressed. Why not? Because Judges will bend over backwards to ensure that they don't piss off police (who will then withdraw support from them, and help run candidates against them, possibly costing them their jobs). So, such a few number of motions to suppress are ever granted, the rules have become almost meaningless.
Bottom line - as nearly meaningless as the rules have become, they still paint very specific rules for the police to follow which protect all of us, criminals or not, every time we come in contact with the state. They ostensibly protect us in our homes, that police can't just break down our door with impunity, so they can't pull us over for a minor traffic infraction, take us the station, and search our cars top to bottom for no reason.
On the other hand, the rules are strict, and the result has probably been that this leads to a greater culture of police fabrication of evidence than any other thing in the law. The problem is, once the line of fabrication has been crossed, so easily and so often, it is hard to ever tell the difference between lying about probable cause to nail a seemingly "guilty" person, and lying about actual evidence to nail that same "guilty" person.
All in all, without it, there will be functionally no check on law enforcement, so when the Supreme Court hears a case challenging the whole basis of the exclusionary rule next year, cross your collective fingers that your freedoms remain somewhat intact, and the rule remains in place.