There have been bit-addressable computers, with variable-sized words (up to a limit), see (for example) patent US4467443.
However, the more assumptions you can make and the less flexibility you have, the faster you can make your RAM. If you know that you will only ever fetch bytes (or 16-bit words, or 32-bit words, or 16-byte memory lines), you can wire up all of this in parallel, but that is harder (although not impossible) to do with a more flexible design.
There's also been CPUs where the size of the word fetched decides the granularity of the addressing. So bytes can be at any byte-address, 16-bit words must be at an even address, and 32-bit words at an address that is a multiple of four.