Introduction
------------

Parallel Architectures

Multicomputer, an idealized parallel computer
- Interconnection network
- Node
  - CPU
  - Memory

Distributed Memory Computer (MPP)
Shared Memomory Computer (SMP)
Local Area Network (Distributed System)

Main issue
- ratio computation/communication performance
- 1 message exchange equivalent to 10000 computation operations

Parallel Programming

- Concurrency
  - Multiple processes (threads of control) at same time
- Interaction
  - Communication: exchange of information
  - Synchronization: waiting for common time

Tradeoff
- Parallelism
  - speedup by many concurrent processes
- Locality
  - local execution more efficient
  - process creation overhead
  - communication/synchronization overhead

Performance Issues
- Partitioning
  - which activities to spawn as separate processes
  - which activiities to execute within process
- Mapping
  - where to start separate processes
  - local node, neighbor node, non-neighbor node
- Load balancing
  - all nodes should be constantly busy with work
  - many small processes preferred
  - conflict with locality

Software engineering issues
- Basic approaches
  - Development of parallel program
  - Parallelization of sequential program
- Portability
  - Program independent of actual kind of architecture
  - Program independent of number of processors 
- Scalability
  - Program should be suitable for growing(shrinking) number of processors
- Determinism
  - Program should always yield same result