One of the main difficulties in developing parallel applications is getting data access right and avoiding race conditions or situations that would corrupt shared data. Sometimes, these situations are easy to spot as they lead to spectacular crashes. Other times, more worryingly, they are not—the application keeps plodding along, producing incorrect results.

It is always important to have good tests for our applications and their internal functions. It is even more so for parallel applications, where building a clear mental picture of what happens where and when can be particularly challenging.

Another difficulty in parallelizing algorithms is to know when to stop. Amdahl's law tells us very clearly that parallelization is, from a given point forward, a game of diminishing returns. Experience tells us that parallelization efforts can easily become a time sink. It is important to have clarity in our minds as to which parts of the code need to be parallelized (if...