To get an OS up and running, there are various things being executed which are typically called "loaders" and "bootstrappers".
Their collective purpose is exactly as their names imply: to get the OS up and running, one step at a time.
Some of these executable code are stored in read-only memory (ROM), or some type of reprogrammable memory (such as Flash ROM), or a certain predetermined location on the storage disk (located on a so-called "boot sector", which is a predetermined location on a "disk partition") from which the machine can be started up.
Your suspicion that at least some of these executable code must exist outside the concept we call "files" or "file system" is correct. Not all of them are; but at least some of them must be.
The computer hardware-firmware, and operating system itself typically are able to access these "non-file" storage areas to perform infrequent maintenance upgrades to these executable code.
The purpose of allowing these upgrades from software is an engineering choice driven by the consumer market - it is cheaper to maintain computers if is it possible for customers to perform the upgrades, than to require machines to be shipped back to a factory or maintenance center to do the same.
Are these executable code different from the code found in "executable files"? Maybe, maybe not. There are certainly some differences.
A 64-bit machine may spend 99.99% of its time executing 64-bit machine code once booted; but during the initial seconds it may have to execute some 32-bit machine code.
Likewise, a fully-loaded OS will execute in privileged mode and the rest of the code execute in user mode. But during bootstrapping most of the code may execute in privileged mode - or rather, in some "simply unprotected" mode, because the CPU mechanism for software fault handling may not have been fully initialized.
Code that needs to execute without the OS, obviously cannot make OS calls. They can still make jumps and subroutine calls to other code - as long as they are located within the same piece of code that is loaded into RAM in one swoop.
This brings up the "size limitation" of bootstrapper code. The previous stage has to know exactly how big is the code size of the next stage. That amount of executable code/data would be brought into RAM before any of it can be executed. In the absence of an OS (and also of any memory-management mechanism), the bootstrapper cannot make subroutine calls located outside what has been brought into RAM - unless if the bootstrapper knows how to load that the extra code.
The BIOS may have to provide a way to facilitate (simplify) the loading of extra data from the storage disk. Performance or security is not the concern - the only concern is simplicity for the caller (which is the bootstrapper). Otherwise, that would be a very tedious task, and would eat up the precious space the bootstrapper code has to fit in.
What formats are used for such bootstrapping code?
These code must follow some kind of conventions in order to work correctly. Whether these conventions qualify as "a format" is a word-play.
For example, the starting offset of the first machine instruction in the bootstrapping code must be specified. Otherwise, how else do you start executing it?
Some bootstrapping code are required to carry a checksum, or even a digital signature (i.e. become digitally signed), to prevent trying to execute a bootstrapper whose data is corrupted, or spoofed or tampered with by malicious intent.
These conventions must be agreed upon by the machine manufacturers, operating systems vendors and/or maintainers, and so on.