There are two popular approaches to building applications that benefit from multiple CPUs:
The application must be written to be multi-threaded.
The application is packaged into small components, and placed in containers that take advantage of multiple CPUs.
You may recognize the first approach as your typical monolithic application, while the second leans towards Service-Oriented Architecture (SOA).
Each approach has it’s advantages and disadvantages. The first approach is popular with developers, as the developer has direct control of the system (whether virtual or not). However, it is less popular with IT Operations because big monolithic applications are harder to manage, and may hog the resources of the system.
In comparison, SOA breaks things down into small lego-sized pieces, and then assembles them together into something useful. This is not a new concept, rather it is as old as the UNIX shell itself. Whenever you use a pipe (|) in the shell, you are effectively taking small programs and connecting them together into something useful. For example:
ls | grep '\.txt' | sort
The component architecture that is used above is the text input and output of the UNIX shell. Because each command depends on the previous command for input, there is little benefit with multiple CPUs. However, programs that take a long time to complete can be assigned a different CPU, resulting in smoother operations.
The potential disadvantage of SOA is that you may end up having multiple pieces, but not a coherent strategy or way of stitching the pieces together. Component architecture and testing is something that should be worked out beforehand. The component architecture defines how these components communicate. Functional tests ensure that each piece behaves as expected. With functional tests, you should know exactly what you’re going to get based on what you put in.
There are other ways to take advantage of multiple CPUs, but the principles are the same. Those are: manage concurrency manually, or break things apart to manageable pieces and let the container (AKA: process, actor, isolate, etc.) manage the CPUs.