We usually have a lot of system software inside our PC's like Operating Systems (which contains different interrupt handlers etc.), Assemblers and Compilers, Drivers. My question is how do we store this information in the computer? Are these all softwares stored in machine instruction format inside the computer at specific locations inside the PC-ROM? Or they are stored as is in the high-level language (like C) in which they are written and then compiled every time the programs are called?
The processors of computers can only understand instructions of their specific machine language. So as you guessed, most of the programs (including the operating system itself) are stored in machine language format on a hard disk or other storage device, or in the permanent EPROM memory of the computer. When it is required, the program code is loaded into memory and then it can be executed.
There are some programs (usually called "scripts") though which are stored in source code format on the disk and then interpreted by a suitable compiler when there is a need to run them. However, the code of the compiler itself must be directly executable by the system, so it must be in machine code.
There exists a cross between these two options as well, called bytecode. In this case, the compiler transforms the original, human-readable source code into a shorter and lower level binary format which is not human readable anymore, but not directly interpretable to a physical processor either. So this requires another compiler or interpreter in a special environment to execute it on a physical device. This special environment is called a virtual machine. This is how Java or C# works, for example.
The advantage of this compared to compiling directly to machine code is that the machine code is specific to a processor family, so it can only run on computers with the right kind of processors in them. Whereas the bytecode can be independent of any specific processor architecture or machine language format, so it is runnable on many different kinds of computers, using the virtual machine variant specific to that type of computer.
And the advantage over interpreted scripts is that the bytecode is much more concise and standardized, so compiling it further to machine code is simpler and much faster than interpreting scripts. Moreover, the bytecode compiler can already verify the code's correctness to a significant extent way before deploying it to a physical machine to run it.
If you want to understand better how computers are built from the ground up, I recommend the book CODE by Charles Petzold.