Since you want to learn how lexers work, I presume you actually want to know how lexer generators work.
A lexer generator takes a lexical specification, which is a list of rules (regular-expression-token pairs), and generates a lexer. This resulting lexer can then transform an input (character) string into a token string according to this list of rules.
The method that is most commonly used mainly consists of transforming a regular expression into a deterministic finite automata (DFA) via a nondeterministic automata (NFA), plus a few details.
A detailed guide of doing this transformation can be found here. Note that I haven't read it myself, but it looks quite good. Also, just about any book on compiler construction will feature this transformation in the first few chapters.
If you are interested in lecture slides of courses on the topic, there are no doubt an endless amount of them from courses on compiler construction. From my university, you can find such slides here and here.
There are few more things that are not commonly employed in lexers or treated in texts, but are quite useful nonetheless:
Firstly, handling Unicode is somewhat nontrivial. The problem is that ASCII input is only 8 bits wide, which means that you can easily have a transition table for every state in the DFA, because they only have 256 entries. However, Unicode, being 16 bits wide (if you use UTF-16), requires 64k tables for every entry in the DFA. If you have complex grammars, this may start taking up quite some space. Filling these tables also starts taking quite a bit of time.
Alternatively, you could generate interval trees. A range tree may contain the tuples ('a', 'z'), ('A', 'Z') for example, which is a lot more memory efficient than having the full table. If you maintain non-overlapping intervals, you can use any balanced binary tree for this purpose. The running time is linear in the number of bits you need for every character, so O(16) in the Unicode case. However, in the best case, it will usually be quite a bit less.
One more issue is that the lexers as commonly generated actually have a worst-case quadratic performance. Although this worst-case behaviour is not commonly seen, it might bite you. If you run into the problem and want to solve it, a paper describing how to achieve linear time can be found here.
You'll probably want to be able to describe regular expressions in string form, as they normally appear. However, parsing these regular expression descriptions into NFAs (or possibly a recursive intermediate structure first) is a bit of a chicken-egg problem. To parse regular expression descriptions, the Shunting Yard algorithm is very suitable. Wikipedia seems to have an extensive page on the algorithm.