Solving Iteration Problems - When an Iteration Procedure "Hangs Up"

If you are far away from the solution point, several things can go wrong. Besides calculating the direction to step the parameters, the program also has to decide how far to step. As the parameters near the solution point, the discrepancy function graph is usually rather flat and smooth, and the program can gauge very accurately how far to step. However, when the parameters are far from the correct solution, both the step direction and step size may be inaccurate.  In that case, the discrepancy function may actually increase with the first new set of parameter values the program tries. Usually both the step direction and step length are in error, but if the program takes a smaller step in the same direction, the discrepancy function will decrease. Consequently, the program starts trying out smaller steps in the same direction. At this stage, the program is performing a line search. There are several line search algorithms available in SEPATH. In any particular situation, one may work where others fail.

The way line search algorithms work is they multiply the step increment vector by a constant (usually less than 1), use this altered vector to recompute new parameter values, and recompute the discrepancy function. If the discrepancy function has not improved, they try again, etc. Hence, when the initial step does not work well, you will see a slowdown in iteration and values of Lambda in the Iteration Results dialog that are less than 1. In some cases, iteration may seem to "hang up" briefly. Ultimately, iteration will cease, with a very small value of Lambda and Steplen.

If the program cannot find a multiple of the step increment vector that reduces the discrepancy function, it means the program has chosen a very bad step direction, or possibly that the program has iterated to a "saddle point."  The program may try reversing the step direction, or may terminate iteration. This kind of situation will generally occur only when the initial values are very far from the solution point, or when the discrepancy function surface is extremely irregular for the particular problem being analyzed.  

There are other telltale signs that indicate that you have iterated into a "bad" region of parameter space.  In computing the step direction, SEPATH computes an approximation to the covariance matrix of the parameters. If the program iterates to a particularly bad region, this matrix may become singular. In some cases, the program can "iterate out" of this problem area by partialling out the parameters that are causing the problem, and continuing the iteration by altering only the other parameters (Jennrich and Sampson, 1968).  

An indication that this has happened is that the NRP value will change during iteration from zero to some other value, indicating that the information matrix has changed from nonsingular to singular. (The NRP value will also be non-zero, generally from the start of iteration, if the model has redundant parameters, i.e., is not identified properly.)

When the program ends up in such an unfortunate situation, there is little you can do but to try again.  

For the maximum likelihood discrepancy function to be computable, both the sample covariance and the covariance matrix reproduced by your model and parameter estimates must be nonsingular. Sometimes during iteration, the program steps to values that yield a singular estimated covariance matrix. If this happens, iteration will terminate immediately.  Generally, this problem will occur only in the early stages of iteration.

If iteration hangs due to any of the above conditions, there are several things to try:

  1. If iteration failed using arbitrary initial values (the default), try the Automatic option for Initial Values in the Analysis Parameters dialog box. These values may get closer to a solution than arbitrary starting values, especially when the covariance matrix is ill-scaled, i.e., has variables that vary widely in variance.
  2. Often, a large step is taken on the first few iterations. Sometimes, this step will carry the parameters into a region from which the iterative process cannot recover. Consequently, one thing to try is to reduce the step size the program is allowed to take. Try reducing the step size to a small value (like .1) and see what happens. Keep in mind that iteration to a solution will take longer under these conditions. During the early phases of iteration, the program will indicate that the largest allowable step size has been taken by placing an asterisk next to the StepLen value printed in the Iteration dialog window.
  3. Sometimes iteration will fail with one line search algorithm but succeed with another. Try an exact (Golden Search) line search by selecting this option in the Line Search Method group in the Analysis Parameters dialog.
  4. When iteration encounters problems early, it is often because the approximate Hessian is too inaccurate far from the solution to be of use in calculating a proper step direction. In such cases, inserting steepest descent iterations at the beginning of iteration will eliminate the problem. Steepest descent iterations use only first derivative information in selecting a step direction, and do not employ the approximate Hessian calculated by the Gauss-Newton procedure.  
  5. If you are testing your model on the covariance matrix, try testing it on the correlation matrix instead. In many situations, models that fail to iterate properly with an ill-scaled covariance matrix will work fine when tested on the correlation matrix.
  6. Keep in mind that you may need to try several of the above options in combination to produce successful iteration. Feel free to experiment.
  7. Remember that a badly miss-specified model may fail to fit properly. One of the common mistakes beginners make is to forget to add disturbance terms to endogenous latent variables. The Structural Modeling Wizard will add these for you automatically, but if you are copying a model from a path diagram, remember that some authors consider disturbance terms to be "implicitly obvious," and don't bother to put them in the diagram itself.