Calculating CRCs (or, better, sha1sums) on both files requires reading every byte anyway. If you do a byte-by-byte comparison, you can quit as soon as you see a mismatch -- and you don't have to worry about two different files that happen to have the same checksum (though that's vanishingly unlikely for sha1sum in the absence of deliberate collisions, which became feasible after I originally wrote this answer). So if you're doing the comparison locally, a byte-by-byte comparison will be at least as fast as a checksum comparison (unless you've already computed the checksums anyway).
On the other hand, checksum comparisons are useful when you're comparing files that aren't on the same machine; the checksums can be computed locally, and you don't have to transfer the entire content over the network.
Hybrid approaches are also possible. For example, you might compute and compare checksums for the two files a chunk at a time, which can avoid reading the whole files (if they differ) while also avoiding transmitting the whole file across the network. The rsync protocol does something like this.
Note that using a simple CRC gives you a fair chance of a collision, as Dave Rager mentioned in his answer. Use at least sha1sum, or even something more recent. (Don't try to invent your own hashing algorithm; the folks who developed sha1sum know far more about this stuff than either of us.)
As for the likelihood of collision, if you use a decent hash like sha1sum you pretty much don't have to worry about it, unless someone is deliberately and expensively constructing files whose sha1sums collide (generating such collisions was not feasible when I first wrote this, but progress is being made). Quoting Scott Chacon's "Pro Git", section 6.1:
Here’s an example to give you an idea of what it would take to get a
SHA-1 collision. If all 6.5 billion humans on Earth were programming,
and every second, each one was producing code that was the equivalent
of the entire Linux kernel history (1 million Git objects) and pushing
it into one enormous Git repository, it would take 5 years until that
repository contained enough objects to have a 50% probability of a
single SHA-1 object collision. A higher probability exists that every
member of your programming team will be attacked and killed by wolves
in unrelated incidents on the same night.
Summary :
Byte-by-byte comparison is good for local comparisons. sha1sum is good for remote comparison, and presents no significant chance of accidental false positives.
And there are newer checksum checksum algorithms that are (so far) less vulnerable to deliberate collisions than SHA1 is. I won't be too specific to avoid having this answer become obsolete yet again in a few years.
sha1sum
you pretty much don't have to worry about it, unless someone is deliberately and expensively constructing files whose sha1sums collide. I don't have a source for this, but I've heard (in the context of git) that the probability of two different files having the same sha1sum is about the same as the probability of every member of your development team being eaten by wolves. On the same day. In completely unrelated incidents.