Basically, a computer is a processor (CPU), some memory, and some peripherals.
The processor is the core component of the computer. The CPU gets data from memory, computes it and put it back into memory.
The processor can receive signals, like reset signal or interrupts signals. It also gets a periodic signal from a clock that tells it when to perform cycles (an instruction is "made" of several cycles).
Except signals, the only communication between the processor and the rest of the world goes through the memory. As different types of memory are available (RAM, ROM or what so ever), a memory bus is in charge of interfacing the processor to these types of memory. Depending of the address given by the processor, the memory bus forwards request to the right memory.
To drive peripherals, the processor uses the memory (once again, the only gate to the outside world). A specific type of memory, the I/O Unit is connected to the memory bus. The I/O Unit is not really memory as it does not store information. When an address is accessed in the IOU, it activates something in a peripheral or read some information from it. For example, accessing address 15 could activate a light or reading from address 17 could read a button status. That's basically the way processor interacts with peripherals.
If we put that all together, we have a "simple" schema of what is a computer :
The orange box, containing CPU, memory and IOU but neither peripherals nor clock, represents what I called the "core". The core is the minimal component necessary for the computer to work. Peripherals only connect the computer to the world and, in particular, the user but they are not necessary for computing. I've put the clock outside because I consider the clock as an interface between the computer and the real time. We can change the clock frequency without affecting the computer behavior. We could even use a manual clock to study a computer step by step.