Each loop list entry contains the following fields:

The icon at the start of each line summarizes briefly the following information:

To understand the meaning of the various icons, choose Admin > Icon Legend... .

To resize the loop list display and provide more room in the main window for loop information, use the adjustment button. The adjustment button is a small square below the Previous Loop button and just above the vertical scroll bar on the right side of the loop information display. In many of the following figures, the loop list is resized from its original configuration.

The loop list Search field allows you to find occurrences of any character in the loop list. You can search for subroutine names, a phrase (such as parallel or region), or Olid numbers. (See Figure 2-2.)

The search is not case sensitive; simply key in the string. To find subsequent occurrences of the same string, press the Enter key.

You can sort the loop list either in the order of the source code, or by performance cost (if you are running the WorkShop performance analyzer). You usually control sorting with the sort option button, the left-most button below the Search field.

When loops are sorted in source order, the loop-ID is indented according to the nesting level of the loop. For the demonstration program, only the last several loops are nested, so you have to scroll down to see indented loop-IDs. For example, scroll down the loop list until you find a loop whose nest value, as shown in the loop list, is greater than 2.

When loops are sorted by performance cost, using Sort by Perf.Cost option button, the list is not indented. The sorting option is grayed out in the example because the performance analyzer is not currently running.

You may want to look at only some of the loops in a large program. The loop list can be filtered according to parallelization status and loop origin. The filter parameters are controlled by the two option buttons to the right of the sort option button.

Procedure 2-1. Filtering the Loop List by Parallelization State

Filtering according to parallelization state allows you to focus, for example, on loops that were not automatically parallelized by the compiler but that might still run concurrently if you add appropriate directives.

Filtering is controlled by the Show Loop Types option button centered below the loop list; the default setting is Show All Loop Types, as shown in Figure 2-3.

Figure 2-3. Show Loop Types Option Button

You can select according to the following states of loop parallelization and processing (which are displayed when you click the show loop types option button):

• Show All Loop Types, the default.

• Show Unparallelizable Loops displays loops that are running serially because they could not be parallelized.

• Show Parallelized Loops displays loops that were parallelized.

• Show Serial Loops displays loops that are best run serially.

• Show PCF Directives displays loops containing PCF directives.

• Show OMP Directives displays loops containing OpenMP directives.

• Show Modified Loops displays loops for which modifications have been requested.

The second, third, and fourth categories correspond to parallelization icons in the Icon Legend... window. Making modifications to loops is described in “Making Changes”.

To see the effects of these options, choose them in turn by clicking on the option button and selecting each option. If you choose the Show Modified Loops option, a message appears that no loops meet the filter criterion, because you have not made any modifications.

To see detailed information about the transformed loops derived from a particular loop, pull down the Views > Transformed Loops View option.

Figure 2-9. Transformed Loops View for Loop Olid 1

Selecting Transformed Loops

You can also select specific transformed loops. When you click a highlight button in the Transformed Loop View, the highlighting of the original source typically changes color, although for loop Olid 1 the highlighted lines do not (see Figure 2-10). For loops with more extensive transformations, the set of highlighted lines is different when you select from the Transformed Loops View (for example, see “Fused Loops”).

Transformed loops can also be selected by clicking the corresponding loop brackets in the Transformed Source window.

Figure 2-10. Transformed Loops in Source Windows

You can either leave the Transformed Loops View open or close it by selecting its Admin > Close menu item. When looking at subsequent loops, you might find it useful to see the information in the Transformed Loops View.

Scroll to the top of the list of loops and select loop Olid 2. This loop is a simple loop: computations in each iteration are independent of each other. It was transformed by the compiler to run concurrently. Notice in the Transformed Source window the directives added by the compiler.

      DO 1000 I = 1, NSIZE
          A(I) = B(I)*C(I)
1000   CONTINUE

Move to the next loop by clicking the Next Loop button.

Olid 2 is a simple loop with too little content to justify running it in parallel. The compiler determined that the overhead of parallelizing would exceed the benefits; the original loop and the transformed loop are identical.

      DO 1100 I = 1, NSIZE
        A(I) = B(I)*C(I)
1100  CONTINUE

Move to the next loop by clicking the Next Loop button.

Loop Olid 3 is parallelized because it contains an explicit C$OMP PARALLEL DO directive in the source, as is shown in the loop information display (Figure 2-11). The compiler passes the directive through to the transformed source.

The loop parallelization status option button is set to C$OMP PARALLEL DO..., and it is shown with a highlight button. Clicking the highlight button brings up both the Source View (Figure 2-12), if it is not already opened, and the Parallelization Control View, which shows more information about the parallelization directive.

Figure 2-11. Explicitly Parallelized Loop

If you clicked on the highlight button, close the Parallelization Control View. (Using the Parallelization Control View is discussed in “Adding C$OMP PARALLEL DO Directives and Clauses”.)

Figure 2-12. Source View of C$OMP PARALLEL DO Directive

Close the Source View and move to the next loop by clicking the Next Loop button.

Loops Olid 5 and Olid 6 are simple parallel loops that have similar structures. The compiler combines these loops to decrease overhead. Note that loop Olid 6 is described as fused in the loop information display and in the Transformed Loops View; it is incorporated into the parallelized loop Olid 5. If you look at the Transformed Source window and select Olid 5 and Olid 6, the identical lines of code are highlighted for each loop.

        DO 1300 I = 1, NSIZE
        A(I) = B(I) + C(I)
1300    CONTINUE        
        DO 1350 I = 1, NSIZE
        AA(I,NSIZE) = B(I) + C(I)
1350    CONTINUE        

Move to the next loop by clicking Next Loop twice.

Loop Olid 7 is an example of a loop that the compiler can eliminate entirely. The compiler determines that the body is independent of the rest of the loop. It moves the body outside of the loop, and eliminates the loop. The transformed source is not scrolled and highlighted when you select Olid 7 because there is no transformed loop derived from the original loop.

        DO 1500 I = 1, NSIZE
        XX = 10.0
1500    CONTINUE        

Move to the next loop, Olid 8, by clicking the Next Loop button. This loop is discussed in “Unparallelizable Carried Data Dependence”.

Carried data dependence typically arises when recurrence of a variable occurs in a loop. Depending on the nature of the recurrence, parallelizing the loop may be impossible. The following loops illustrate four kinds of data dependence:

        DO 2000 I = 1, NSIZE-1
        A(I) = A(I+1)
2000    CONTINUE        

Parallelizable Carried Data Dependence

Loop Olid 9 has a structure similar to loop Olid 8. Despite the similarity however, Olid 9 may be parallelized.

Example 2-6. Parallelizable Carried Data Dependence

C*$*ASSERT DO (CONCURRENT) DO 2100 I = 1, NSIZE A(I) = A(I+M) 2100 CONTINUE

Note that the array indices differ by offset M. If M is equal to NSIZE and the array is twice NSIZE, the code is actually copying the upper half of the array into the lower half, a process that can be run in parallel. The compiler cannot recognize this from the source, but the code has the assertion C*$* ASSERT DO (CONCURRENT) so the loop is parallelized.

Click the highlight button to show the assertion in the Source View.

Figure 2-13. Parallelizable Data Dependence

Move to the next loop by clicking the Next Loop button.

Multi-line Data Dependence

Data dependence can involve more than one line of a program. In loop Olid 10, a dependence similar to that in Olid 9 occurs, but the variable is set and used on different lines.

Example 2-7. Multi-line Data Dependence

DO 2200 I = 1, NSIZE-1 B(I) = A(I) A(I+1) = B(I) 2200 CONTINUE

Click the highlight button on the obstacle line.

In the Source View, highlighting shows the dependency variable on the two lines (see the figure that follows). Of course, real programs, typically, have far more complex dependences than this.

Move to the next loop by clicking Next Loop.

Figure 2-14. Highlighting on Multiple Lines

        DO 2300 I = 1, NSIZE
        X =  B(I)*C(I) + X
2300    CONTINUE        

Loop Olid 12 has an input/output (I/O) operation in it. It cannot be parallelized because the output would appear in a different order depending on the scheduling of the individual CPUs.

        DO 2500 I = 1, NSIZE
        print 2599, I, A(I)
2599        format("Element A(",I2,") = ",f10.2)
2500    CONTINUE        

Click the button indicating the obstacle, and note the highlighting of the print statement in the Source View.

Move to the next loop by clicking Next Loop.

Loop Olid 13 has an unstructured control flow: the flow is not controlled by nested if statements. Typically, this problem arises when goto statements are used; if you can get the branching behavior you need by using nested if statements, the compiler can better optimize your program.

        DO 2600 I = 1, NSIZE
        A(I) = B(I)*C(I)
        IF (A(I) .EQ. 0) GO TO 2650
2600    CONTINUE        

Because the goto statement is essential to the program's behavior, the compiler cannot determine how many iterations will take place before exiting the loop. If the compiler parallelized the loop, one thread might execute iterations past the point where another has determined to exit.

Click the highlight button in the Obstacles to Parallelization information block in the loop information display, next to the unstructured control flow message. Note that the line with the exit from the loop is highlighted in the Source View.

Move to the next loop by clicking Next Loop.

Unless you make an assertion, a loop with a subroutine call cannot be parallelized; the compiler cannot determine whether a call has side effects (such as creating data dependencies.)

        DO 2700 I = 1, NSIZE
        A(I) = B(I) + RTC()
2700    CONTINUE        

C*$*ASSERT CONCURRENT CALL
        DO 2800 I = 1, NSIZE
        A(I) = B(I) + FOO()
2800    CONTINUE

If you specify array index values by values in another array (referred to as a permutation vector), the compiler cannot determine if the values in the permutation vector are distinct. If the values are distinct, loop iterations do not depend on each other and the loop can be parallelized; if they are not, the loop cannot be parallelized. Thus, without an assertion, a loop with a permutation vector is not parallelized.

        DO 3200 I = 1, NSIZE-1
        A(IC(I)) = A(IC(I)) + DELTA
3200    CONTINUE        

C*$*ASSERT PERMUTATION(ib)
        DO 3300 I = 1, NSIZE
        A(IB(I)) = A(IB(I)) + DELTA
3300    CONTINUE        

All of the messages that can be found in an Obstacles to Parallelization information block are found in the following lists. Because they include specific loop and line information, messages that appear in the loop information display differ slightly from those in the tables.

The following table lists the Obstacles to Parallelization block messages that deal with dependence issues (such as those involving scalars, arrays, missing information, and finalization).

subroutine fun02(a, b, n, s)
    integer a(n), b(n), s, n
    do i = 1, n
      if (a(i) .gt. 0) then
        s = a(i)
      end if
      b(i) = a(i) + s
    end do
    end

       subroutine fun04(a, b, n)
        integer i, j, k, n
        integer b(n), a(n,n,n)
        do i = 1, n
          do j = i + 3, n
c*$* no fusion
            do k = 1, n
              b(k) = k
            end do
            do k = 1, n
              a(i,j,k) = a(i,j,k) + b(k)
            end do
          end do
        end do
        end

Loop Olid 18 is the outer loop of a pair of loops and it runs in parallel. The inner loop runs in serial, because the compiler knows that one parallel loop should not be nested inside another. However, you can force parallelization in this context by inserting a C$OMP PARALLEL DO directive with the C$SGI&NEST clause. For example, see “Distributed and Reshaped Arrays: C$SGI DISTRIBUTE_RESHAPE”.

        DO 4000   I = 1,NSIZE
          DO 4010   J = 1,NSIZE
            AA(J,I) = BB(J, I)
4010      CONTINUE        
4000    CONTINUE        

Click Next Loop to move to the inner loop, Olid 19.

Notice that when you select the inner loop that the end-of-loop continue statement is not highlighted. This happens for all interior loops and is a compiler error that disrupts line numbering in the Parallel Analyzer View. Be careful if you use the Parallel Analyzer View to insert a directive for an interior loop; check that the directive is properly placed in your source code.

Click Next Loop again to select the outer loop of the next nested pair.

The outer loop, Olid 20, is shown in the loop information display as a serial loop inside a parallel loop. The original interior loop is labelled as parallel, indicating the order of the loops has been interchanged. This happens because the compiler recognized that the two loops can be interchanged, and that the CPU cache is likely to be more efficiently used if the loops are run in the interchanged order. Explanatory messages appear in the loop information display.

        DO 4100   I = 1,NSIZE
          DO 4110   J = 1,NSIZE
            AA(I,J) = BB(I, J)
4110      CONTINUE        
4100    CONTINUE        

Move to the inner loop, Olid 21, by clicking the Next Loop button.

Click Next Loop once again to move to the following triply-nested loop.

The order of Olid 22 and Olid 23 has been interchanged. As with the previous nested loops, the compiler recognizes that cache misses are less likely.

        DO 5000   I = 1,NSIZE
          DO 5010   J = 1,NSIZE
            CC(I,J) = 0.
            DO 5020   K = 1,NSIZE
              CC(I,J) = CC(I,J) + AA(I,K)* BB(K,J)
5020        CONTINUE        
5010      CONTINUE        
5000    CONTINUE        

Double-click on Olid 22, Olid 23, and Olid 24 in the loop list and note that the loop information display shows that Olid 22 and Olid 24 are serial loops inside a parallel loop, Olid 23.

Because the innermost serial loop, Olid 24, depends without recurrence on the indices of Olid 22 and Olid 23, iterations of loop Olid 22 can run concurrently. The compiler does not recognize this possibility. This brings us to the subject of the next section, the use of the Parallel Analyzer View tools to modify the source.

Return to Olid 22, if necessary, by using the Previous Loop button.

You make changes by one of the following actions:

You can request changes in any order; there are no dependencies implied by the order of requests. The following changes are discussed in this section:

Adding C$OMP PARALLEL DO Directives and Clauses

Loop Olid 22, shown in Figure 2-15, is a serial loop nested inside a parallel loop. It is not parallelized, but its iterations could run concurrently.

To add a C$OMP PARALLEL DO directive to Olid 22, do the following:

1. Make sure loop Olid 22 is selected.

2. Click on the loop parallelization status option button (Figure 2-15) and choose C$OMP PARALLEL DO... to parallelize Olid 22.

This sequence requests a change in the source code, and opens the Parallelization Control View (Figure 2-15). You can now look at variables in the loop and attach clauses to the directive, if needed.

Figure 2-15. Requesting a C$OMP PARALLEL DO Directive

Figure 2-16, shows information presented in the Parallelization Control View for a C$OMP PARALLEL DO directive. (For the C$OMP DO directive, see “Parallelization Control View” in Chapter 6):

• The selected loop.

• Condition for parallelization editable text field.

• MP scheduling option button.

• MP Chunk size editable text field.

• PRIVATE, SHARED, DEFAULT, FIRSTPRIVATE, LASTPRIVATE, COPYIN, REDUCTION, AFFINITY, NEST, and ONTO clause windows.

• A list of all the variables in the loop, each with an icon indicating whether the variable was read, written, or both; these icons are introduced in “Loop List Icons”.

In the list of variables, each variable has a highlight button to indicate in the Source View its use within the loop; click some of the buttons to see the variables highlighted in the source view. After each variable's name, there is a descriptor of its storage class: Automatic, Common, or Reference. (See “Variable List Storage Labeling” in Chapter 6.)

You can add clauses to the directive by placing appropriate parameters in the text fields, or using the options menus.

Figure 2-16. Parallelization Control View After Choosing C$OMP PARALLEL DO...

Notice that in the loop list, there is now a red plus sign next to this loop, indicating that a change has been requested (see Figure 2-17).

Figure 2-17. Effect of Changes on the Loop List

Close the Parallelization Control View by using its Admin -> Close option.

Deleting Assertions or Directives

Move to the next loop, Olid 15 (shown in Example 2-12).

To delete an assertion, follow these steps:

1. Find the assertion C*$* ASSERT CONCURRENT CALL in the loop information display.

2. Select its Delete option button.

Figure 2-18 shows the state of the assertion in the information display. A similar procedure is used to delete directives.

Figure 2-18. Deleting an Assertion

From this point, the next non-optional step in the tutorial is at the beginning of “Updating the Source File”.

Now you have requested a set of changes. Using the controls in the Update menu, you can update the file. These are the main actions that the Parallel Analyzer View performs during file modification:

Viewing Changes With gdiff

By default, the Parallel Analyzer View does not open a gdiff window. To open a gdiff window that shows the requested changes to the source file before compiling the modified code, toggle the checkbox in Update -> Run gdiff After Update (Figure 2-19).

Figure 2-19. Run gdiff After Update

If you always wish to see the gdiff window, you can set the resource in your .Xdefaults file:

cvpav*gDiff: True

cvpav*runUserEdit: True

cvpav*userEdit: xwsh -e vi %s +%d

Updating the Source File

Choose Update  -> Update All Files to update the source file to include the changes requested in this tutorial. (See .) Alternatively, you can use the keyboard shortcut for this operation, Ctrl+U, with the cursor anywhere in the main view.

If you have set the checkbox and opened the gdiff window or an editor, examine the changes or edit the file as you wish. When you exit these tools, the Parallel Analyzer View spawns the WorkShop Build Manager (Figure 2-20).

Figure 2-20. Build View of Build Manager

---

Note: If you edited any files, verify when the Build Manager comes up that the directory shown is the directory in which you are running the sample session; if not, change it.

---

Click the Build button in the Build Manager window, and the Build Manager reprocesses the changed file.

When the build completes, the Parallel Analyzer View updates to reflect the changes that were made. You can now examine the new version of the file to see the effect of the requested changes.

The first construct in subroutine ompdummy() is a parallel region containing two loops that are explicitly parallelized with C$OMP DO directives. With this construct in place, the loops can execute in parallel, that is, the second loop can start before all iterations of the first complete.

C$OMP PARALLEL SHARED(a,b)
C$OMP DO SCHEDULE(DYNAMIC, 10-2*2)
        DO 6001 I=-100,100
           A(I) = I
6001   CONTINUE
C$OMP DO SCHEDULE(STATIC)
        DO 6002 I=-100,100
           B(I) = 3 * A(I)  
6002   CONTINUE
C$OMP END PARALLEL

Notice in Figure 2-21, that the controls in the loop information display are now labelled Region Controls. The controls now affect the entire region. The Keep option button and the highlight buttons function the same way they do in the Loop Parallelization Controls.

Figure 2-21. Loops Explicitly Parallelized Using C$OMP DO

Click Next Loop twice to step through the two loops. Notice in the Source View that both loops contain a C$OMP DO directive.

Click Next Loop to step to the second parallel region.

The second parallel region, Olid 75, contains a pair of loops that are identical to the previous example except for a barrier between them. Because of the barrier, all iterations of the first C$OMP DO loop must complete before any iteration of the second loop can begin.

C$OMP PARALLEL SHARED(A,B)
C$OMP DO SCHEDULE(STATIC, 10-2*2)
        DO 6003 I=-100,100
           A(I) = I
6003   CONTINUE
C$OMP END DO NOWAIT
C$OMP BARRIER
C$OMP DO SCHEDULE(STATIC)
        DO 6004 I=-100,100
           B(I) = 3 * A(I)  
6004   CONTINUE
C$OMP END PARALLEL

Click Next Loop twice to view the barrier region. (See Figure 2-22.)

Figure 2-22. Loops Using C$OMP BARRIER Synchronization

Click Next Loop twice to go to the third parallel region.

Click Next Loop to view the first of the two loops in the third parallel region. This loop contains a critical section.

C$OMP DO 
       DO 6005 I=1,100
C$OMP CRITICAL(S3)
           S1 = S1 + I
C$OMP END CRITICAL(S3)
6005  CONTINUE

Click Next Loop to view the critical section. The critical section uses a named locking variable (S3) to prevent simultaneous updates of S1 from multiple threads. This is a standard construct for performing a reduction.

Move to the next loop by using Next Loop.

This loop has a single-process section, which ensures that only one thread can execute the statement in the section. Highlighting in the Source View shows the begin and end directives.

       DO 6006 I=1,100
C$OMP SINGLE
           S2 = S2 + I
C$OMP END SINGLE
6006  CONTINUE

Click Next Loop to view information about the single-process section.

Move to the final parallel region in ompdummy() by clicking the Next Loop button.

The fourth and final parallel region of ompdummy() provides an example of parallel sections. In this case, there are three parallel subsections, each of which calls a function. Each function is called exactly once, by a single thread. If there are three or more threads in the program, each function may be called from a different thread. The compiler treats this directive as a single-process directive, which guarantees correct semantics.

C$OMP PARALLEL SHARED(A,C) PRIVATE(I,J)
C$OMP SECTIONS
        call boo
C$OMP SECTION
        call bar
C$OMP SECTION
        call baz
C$OMP END SECTIONS
C$OMP END PARALLEL

Click Next Loop to view the entire C$OMP SECTIONS region.

Click Next Loop to view a C$OMP SECTION region.

Move to the next subroutine by clicking Next Loop twice.

When you select the subroutine dst1d(), a directive is listed in the loop information display that is global to the subroutine. The directive, C$SGI DISTRIBUTE, specifies placement of array members in distributed, shared memory. (See Figure 2-23.)

Figure 2-23. C$SGI DISTRIBUTE Directive and Text Field

In the editable text field adjacent to the directive name is the argument for the directive, which in this case distributes the one-dimensional array a(m) among the local memories of the available processors. To highlight the directive in the Source View, click the highlight button.

Click Next Loop to move to the parallel loop.

The loop has a C$OMP PARALLEL DO directive, which works with C$SGI DISTRIBUTE to ensure that each processor manipulates locally stored data.

        subroutine dst1d(a)

        parameter (m=10)
        real a(m)
C$DISTRIBUTE a(BLOCK)
C$OMP PARALLEL DO
        do i=1,m
          a(i)= i
        end do

        return

You can highlight the C$OMP PARALLEL DO directive in the Source View with either of the highlight buttons in the loop information display. If you use the highlight button in the Loop Parallelization Controls, the Parallelization Control View presents more information about the directive and allows you to change the C$OMP PARALLEL DO clauses. In this example, it confirms what you see in the code: that the index variable i is local.

Click Next Loop again to view the next subroutine.

When you select the subroutine rshape2d(), the subroutine's global directive is listed in the loop information display. The directive, C$SGI DISTRIBUTE_RESHAPE, also specifies placement of array members in distributed, shared memory. It differs from the directive C$SGI DISTRIBUTE in that it causes the compiler to reorganize the layout of the array in memory to guarantee the desired distribution. Furthermore, the unit of memory allocation is not necessarily a page.

In the text field adjacent to the directive name is the argument for the directive, which in this case distributes the columns of the two-dimensional array b(m,m) among the local memories of the available processors. To highlight the directive in the Source View, click the highlight button.

Click the Next Loop button to move to the parallel loop.

The loop has a C$OMP PARALLEL DO directive (Example 2-24), which works with C$SGI DISTRIBUTE_RESHAPE so that each processor manipulates locally stored data.

        subroutine rshape2d(b)
        parameter (m=10)
        real b(m,m)

C$DISTRIBUTE_RESHAPE b(*,BLOCK)
C$OMP PARALLEL DO
C$SGI&NEST (i,j)
        do i=1,m
           do j=1,m
              b(i,j)= i*j
           end do
        end do
        return

If you use the highlight button in the Loop Parallelization Controls, the Parallelization Control View presents more information. In this example, it confirms what you see in the code: that the index variable i is local, and that the nested loop can be run in parallel.

If the code had not had the C$SGI&NEST clause, you could have inserted it by supplying the arguments in the text field in the Parallelization Control View. You can use the C$SGI&NEST clause to parallelize nested loops only when both loops are fully parallel and there is no code between either the do-i and do-j statements or the enddo-i and enddo-j statements.

Click Next Loop to move to the nested loop. Notice that this loop has an icon in the loop list and in the loop information display indicating that it runs in parallel.

Click Next Loop to view the next subroutine, prfetch().

Click Next Loop to go to the first loop in prfetch(). The compiler switched the order of execution of the nested loops, Olid 128 and 129. To see this, look at the Transformed Source view.

      subroutine prfetch(a, b, n)

      integer*4 a(n, n), b(n, n)
      integer i, j, n

      do i=1, n
         do j=1, n
C*$*PREFETCH_REF = b(i,j), STRIDE=2,2 LEVEL=1,2 KIND=rd, SIZE=4
              a(i,j) = b(i,j) 
           end do
        end do

Click Next Loop to move to the nested loop. The list of directives in the loop information display shows C*$* PREFETCH_REF with a highlight button to locate the directive in the Source View. The directive allows you to place appropriate portions of the array in cache.