Rounding in Calculations and Conversions

In this section:

DEFINE and COMPUTE

Most computations are done in floating-point arithmetic. Packed fields are converted to D internally, then back to P. Where the operating system supports it, native arithmetic is used for addition and subtraction for either integer or packed (8-byte) formats. Long packed (16-byte) format is converted to extended precision numbers for computation.

When a field with decimal places is computed to an integer field, the decimal places are truncated, and the resulting value is the integer part of the input value.

When a field with decimal places is computed from one format to another, two conversions take place, unless native arithmetic is being used:

First, the field is converted internally to floating-point notation.
Second, the result of this conversion is converted to the specified format. At this point, the rounding algorithm described previously is applied.

Example: Redefining Field Formats

The following example illustrates some differences in the way packed fields, floating-point fields, decimal precision fields, and integer fields are stored and displayed. It also shows database values redefined to a format with a larger number of decimal places.

Master File

FILE=EXAMPLE, SUFFIX=FOC
 SEGNAME=ONLY, SEGTYPE=S1,$
  FIELD=PACKED2,,P9.2,$
  FIELD=DOUBLE2,,D9.2,$
  FIELD=FLOAT2,, F9.2,$
  FIELD=INTEGER,,I9  ,$
  FIELD=MATH2,,  M9.2,$
  FIELD=XMATH2,, X9.2,$

Program to Load Data

This MODIFY creates a file with six fields: a P field with two decimal places, a D field with two decimal places, an F field with two decimal places, an integer field, an M field with two decimal places, and an X field with two decimal places. The same data values are then loaded into each field.

CREATE FILE EXAMPLE
MODIFY FILE EXAMPLE
FIXFORM PACKED2/9 X1 DOUBLE2/9 X1 FLOAT2/9 X1 INTEGER/9 X1
FIXFORM MATH2/9 X1 XMATH2/9
MATCH PACKED2
ON MATCH REJECT
ON NOMATCH INCLUDE
DATA
1.6666666 1.6666666 1.6666666 1.6666666 1.6666666 1.6666666
125.16666 125.16666 125.16666 125.16666 125.16666 125.16666
5432.6666 5432.6666 5432.6666 5432.6666 5432.6666 5432.6666
4.1666666 4.1666666 4.1666666 4.1666666 4.1666666 4.1666666
5.5       5.5       5.5       5.5       5.5       5.5      
106.66666 106.66666 106.66666 106.66666 106.66666 106.66666
7.2222222 7.2222222 7.2222222 7.2222222 7.2222222 7.2222222
END

Report Request

A DEFINE command creates temporary fields that are equal to PACKED2, DOUBLE2, FLOAT2, MATH2, and XMATH2 with redefined formats containing four decimal places instead of two. These DEFINE fields illustrate the differences in the way packed fields, floating-point fields, and decimal precision fields are stored and displayed.

The request prints the values and a total for all six database fields, and for the five DEFINE fields.

DEFINE FILE EXAMPLE
PACKED4/P9.4=PACKED2;
DOUBLE4/D9.4=DOUBLE2;
FLOAT4/D9.4=FLOAT2;
MATH4/M9.4 = MATH2;
XMATH4/X9.4=XMATH2;
END

TABLE FILE EXAMPLE
PRINT PACKED2 PACKED4 DOUBLE2 DOUBLE4 FLOAT2 FLOAT4 MATH2 MATH4 XMATH2 XMATH4 INTEGER
ON TABLE SUMMARIZE
ON TABLE SET STYLE *
GRID=OFF,$
ENDSTYLE
END

The following image shows the resulting output on z/OS:

The following image shows the resulting output on Windows:

In this example, the PACKED2 sum is an accurate sum of the displayed values, which are the same as the stored values. The PACKED4 values and total are the same as the PACKED2 values.

The DOUBLE2 sum looks off by .01 on z/OS and by .04 on Windows. It is not the sum of the printed values but a rounded sum of the stored values. The DOUBLE4 values show that the DOUBLE2 values are actually rounded from the stored values. The DOUBLE4 values and sum show more of the decimal places from which the DOUBLE2 values are rounded.

The FLOAT2 total seems off by .02 on z/OS. Like the DOUBLE2 total, the FLOAT2 total is a rounded total of the stored FLOAT2 values. F fields are not accurate beyond eight digits, as the FLOAT4 column shows.

The integer sum is an accurate total. Like packed fields, the storage values and displayed values are the same.

The MATH2 and XMATH2 sums look off by .01. They are not the sum of the printed values but a rounded sum of the stored values. The MATH4 and XMATH4 values show that the MATH2 and XMATH2 values are actually rounded from the stored values. The MATH4 and XMATH4 values and sum show more of the decimal places from which the MATH2 and XMATH2 values are rounded.

The following request illustrates the difference between floating-point and MATH data types.

DEFINE FILE ROUND1      
DOUBLE20/D32.20=DOUBLE2;
MATH20/M32.20=MATH2;    
END                     
TABLE FILE ROUND1       
PRINT DOUBLE20 MATH20   
ON TABLE SUMMARIZE 
END

The output shows that the double precision floating-point number is not exactly the same as the input values in most cases. It has extra digits for those values that do not have an exact binary equivalent. The math values represent the input values exactly.

                           DOUBLE20                               MATH20    
                           --------                               ------    
             1.66666660000000010911               1.66666660000000000000    
           125.16666000000000025238             125.16666000000000000000    
         5,432.66659999999956198735           5,432.66660000000000000000    
             4.16666660000000010911               4.16666660000000000000    
             5.50000000000000000000               5.50000000000000000000    
           106.66666000000000025238             106.66666000000000000000    
             7.22222220000000003637               7.22222220000000000000    
                                                                            
TOTAL                                                                       
         5,683.05547539999770378927           5,683.05547540000000000000

DEFINE and COMPUTE

DEFINE and COMPUTE may give different results for rounded fields. DEFINE fields are treated like data source fields, while COMPUTE fields are calculated on the results of the display command in the TABLE request. The following example illustrates this difference:

DEFINE FILE EXAMPLE
DEFP3/P9.3=PACKED2/4;
END

TABLE FILE EXAMPLE
PRINT PACKED2 DEFP3
COMPUTE COMPP3/P9.3=PACKED2/4;
ON TABLE SUMMARIZE
END

The following report results:

PAGE     1
 
 
 PACKED2      DEFP3     COMPP3
 -------      -----     ------
    1.67       .417       .417
  125.17     31.292     31.292
 5432.67   1358.167   1358.167
    4.17      1.042      1.042
    5.50      1.375      1.375
  106.67     26.667     26.667
    7.22      1.805      1.805
 
TOTAL
 5683.07   1420.765   1420.767

The DEFP3 field is the result of a DEFINE. The values are treated like data source field values. The printed total, 1420.765, is the sum of the printed DEFP3 values, just as the PACKED2 total is the sum of the printed PACKED2 values.

The COMPP3 field is the result of a COMPUTE. The printed total, 1420.767, is calculated from the total sum of PACKED2 (5683.07 / 4).