mailing a letter is by far the best way to describe this, you have a name, number, street, town, zip code, maybe a suite number or other depending on where you live.
The letter leaves your house with all of this information on it and the post offices pick apart these items first leaving your house then some first level inspection to broadly figure out is it leaving this post office, is it leaving this town, etc. a number of decisions and sorts along the way to get it to the postal person for that street/area and they sort it by house then deliver it.
The processor and the programmer (you indirectly but with a significant amount of help from the operating system) are you in your house, you have some reference material (your address book) of addresses to things (well lets say the operating system does not necessarily you) some of these things are just ram some are peripherals or control and status registers, etc. there is some boundary in the processor or outside where this flatish address space lives, that boundary is your yard between your house and your mailbox on the curb. the address is tossed onto this bus and then the postman takes over, layers of logic pick apart the address to determine where it needs to go back into the chip or out to peripherals or memory on the board.
So generically yes address can point to things in the cpu and to things outside the cpu. Hardware designers generally dont like to make the work excessively difficult so usually one or a chunk of consecutive bits in the address are used to route the transaction. some high bits might determine on chip or off for example and then some next bits down might determine ram from peripherals, and then once routed to a periperhals some more bits down may determine where in the peripheral. there are also generally gaps do an lspci and look at the address ranges, just because a peripheral wants 32MB doesnt mean it has 32mb worth of individual things to talk to, it is just a nice power of 2 for logic to make decisions on once at the peripheral they may be wasteful if they choose with those bits for accessing a small number of things or efficient.
again the operating system, layered itself, has directly or indirectly the knowledge about that board and processor and generally hides it from you, you normally dont just poke a uart register, you instead call a function at the application layer which calls kernel functions which then eventually get to those registers that make the signals wiggle on the uart or whatever interface. mmus if you have one virtualize addresses so that your application or every application may feel like it starts at some address like 0x0000 and has 0x000 to N bytes of memory to itself so that all programs can be compiled the same way but a translation in hardware and software (operating system) convert your 0x1234 address to some other address before it hits that magic boundary where it leaves the house and hits the mailbox in the post office domain where they just follow the rules for delivery same way every time (hardware).