The act of turning on a computer system and causing its operating system to be loaded  is called booting. The name comes from an image of the computer pulling itself up from its bootstraps, but the act itself slightly more realistic.
During bootstrapping, the computer first loads a small piece of code called the bootstrap loader, which in turn loads and starts the operating system. The bootstrap loader is usually stored in a fixed location on a hard disk or a floppy. The reason for this two step process is that the operating system is big and complicated, but the first piece of code that the computer loads must be very small (a few hundred bytes), to avoid making the firmware unnecessarily complicated.
Different computers do the bootstrapping differently. For PCs, the computer (its BIOS) reads in the first sector (called the boot sector) of a floppy or hard disk. The bootstrap loader is contained within this sector. It loads the operating system from elsewhere on the disk (or from some other place).
After Linux has been loaded, it initialises the hardware and device drivers, and then runs init. init starts other processes to allow users to log in, and do things. The details of this part will be discussed below.
In order to shut down a Linux system, first all processes are told to terminate (this makes them close any files and do other necessary things to keep things tidy), then filesystems and swap areas are unmounted, and finally a message is printed to the console that the power can be turned off. If the proper procedure is not followed, terrible things can and will happen; most importantly, the filesystem buffer cache might not be flushed, which means that all data in it is lost and the filesystem on disk is inconsistent, and therefore possibly unusable.
On early computers, it wasn't enough to merely turn on the computer, you had to manually load the operating system as well. These new-fangled thing-a-ma-jigs do it all by themselves.