Bitmasks are terribly old. I haven't been able to find a reference to the first one, but they were certainly popular by the advent of 8-bit processors, and likely were also used in 4-bit processors.
The idea behind bitmasks is to take advantage of bitwise parallelism. A 8 bit computer can do the same bitwise operation to 8 bits at once if they're packed into a single native word (which means it fits in a register).
The name comes from masking, which is a general approach to covering up areas you don't want to interact with. For example, consider this stencil for masking off areas of a wall (the stencil has been moved after painting to show the pattern)
Masks are also used in photography, where they go by the term "dodge" rather than "stencil." You can use a mask to obscure some of the light during printing to lighten an area.
The term is also used directly in photolithography, which is the technique used to make integrated circuits. The mask prevents light from reaching the photoresist painted on the chip, which creates patterns that later lead to the facinating patterns on the chip. (The below image is one of the masks for the Intel 8080A processor, if you're curious)
Likewise, in bit masking, you are selecting the parts of the word you want to operate on, masking off all the rest of the bits. In the example below, I use the "and" operation to mask the input such that only the 3rd, 4th, and 8th bit show through. The rest are "masked" so that they are 0's. The mask I use is
00110001. I show it below with
# representing 0 and
. representing 1 because that makes the visual appearance of the bitmask similar to that of the physical masks above, and I show a "selected bits" row which shows the bits from the output that were not masked out ("selected bits" is not actually a logical operation that happens... the processor really goes right from input AND mask to output in one step, but I think it clarifies the visual image)
Mask ##..###. (aka 00110001)
(selected) 01 1
Input AND Mask 00010001
As I mentioned, bitmasking is terribly old because it increases productivity of the processor dramatically. On a 4 bit processor, it can make the processor 4x faster. On an 8 bit process, or it can make it 8x faster (on bitwise operations alone, of course).
One fascinating use for this is chess engines. The Chess board has 64 squares. Modern engines have 64 bit integers. This is a terribly convenient bit of luck, so chess engines often leverage it. They have so-called "bitboards" which contain the locations of pieces. This lets you do all sorts of optimizations, such as looking for all pawn moves in a single step.