NAG Library f02 - Eigenvalues and Eigenvectors
|
Scope of the Chapter
|
|
This chapter provides functions for various types of matrix eigenvalue problem:
|
standard eigenvalue problems (finding eigenvalues and eigenvectors of a square matrix );
|
|
singular value problems (finding singular values and singular vectors of a rectangular matrix );
|
|
generalized eigenvalue problems (finding eigenvalues and eigenvectors of a matrix pencil ).
|
Functions are provided for both real and complex data.
Additional functions for these problems can be found in Chapter f08 which contains software derived from LAPACK (see Anderson et al. (1999)). However, you should read the introduction to this chapter before turning to Chapter f08, especially if you are a new user.
Chapter f02 contains Black Box functions that enable many problems to be solved by a call to a single function, and the decision trees in Section [Decision Trees] direct you to the most appropriate functions in Chapter f02.
|
|
Background to the Problems
|
|
Here we describe the different types of problem which can be tackled by the functions in this chapter, and give a brief outline of the methods used to solve them. If you have one specific type of problem to solve, you need only read the relevant sub-section and then turn to Section [Recommendations on Choice and Use of Available Functions]. Consult a standard textbook for a more thorough discussion, for example Golub and Van Loan (1996) or Parlett (1998).
In each sub-section, we first describe the problem in terms of real matrices. The changes needed to adapt the discussion to complex matrices are usually simple and obvious: a matrix transpose such as must be replaced by its conjugate transpose ; symmetric matrices must be replaced by Hermitian matrices, and orthogonal matrices by unitary matrices. Any additional changes are noted at the end of the sub-section.
|
Standard Eigenvalue Problems
|
|
Let be a square matrix of order . The standard eigenvalue problem is to find eigenvalues, , and corresponding eigenvectors, , such that
(1)
(The phrase "eigenvalue problem" is sometimes abbreviated to eigenproblem.)
|
Standard symmetric eigenvalue problems
|
|
If is real symmetric, the eigenvalue problem has many desirable features, and it is advisable to take advantage of symmetry whenever possible.
The eigenvalues are all real, and the eigenvectors can be chosen to be mutually orthogonal. That is, we can write
or equivalently:
(2)
where is a real diagonal matrix whose diagonal elements are the eigenvalues, and is a real orthogonal matrix whose columns are the eigenvectors. This implies that if , and .
Equation (2) can be rewritten
(3)
This is known as the eigen-decomposition or spectral factorization of .
Eigenvalues of a real symmetric matrix are well-conditioned, that is, they are not unduly sensitive to perturbations in the original matrix . The sensitivity of an eigenvector depends on how small the gap is between its eigenvalue and any other eigenvalue: the smaller the gap, the more sensitive the eigenvector. More details on the accuracy of computed eigenvalues and eigenvectors are given in the function documents, and in the f08 Chapter Introduction.
For dense or band matrices, the computation of eigenvalues and eigenvectors proceeds in the following stages:
All the above remarks also apply – with the obvious changes – to the case when is a complex Hermitian matrix. The eigenvectors are complex, but the eigenvalues are all real, and so is the tri-diagonal matrix .
|
|
Standard nonsymmetric eigenvalue problems
|
|
A real nonsymmetric matrix may have complex eigenvalues, occurring as complex conjugate pairs. If is an eigenvector corresponding to a complex eigenvalue , then the complex conjugate vector is the eigenvector corresponding to the complex conjugate eigenvalue . Note that the vector defined in equation (1) is sometimes called a right eigenvector; a left eigenvector is defined by
Functions in this chapter only compute right eigenvectors (the usual requirement), but functions in Chapter f08 can compute left or right eigenvectors or both.
The eigenvalue problem can be solved via the Schur factorization of , defined as
where is an orthogonal matrix and is a real upper quasi-triangular matrix, with the same eigenvalues as . is called the Schur form of . If all the eigenvalues of are real, then is upper triangular, and its diagonal elements are the eigenvalues of . If has complex conjugate pairs of eigenvalues, then has 2 by 2 diagonal blocks, whose eigenvalues are the complex conjugate pairs of eigenvalues of . (The structure of is simpler if the matrices are complex – see below.)
For example, the following matrix is in quasi-triangular form
and has eigenvalues 1, , and 3. (The elements indicated by " " may take any values.)
The columns of are called the Schur vectors. For each , the first columns of form an orthonormal basis for the invariant subspace corresponding to the first eigenvalues on the diagonal of . (An invariant subspace (for ) is a subspace such that for any vector in , is also in .) Because this basis is orthonormal, it is preferable in many applications to compute Schur vectors rather than eigenvectors. It is possible to order the Schur factorization so that any desired set of eigenvalues occupy the leading positions on the diagonal of , and functions for this purpose are provided in Chapter f08.
Note that if is symmetric, the Schur vectors are the same as the eigenvectors, but if is nonsymmetric, they are distinct, and the Schur vectors, being orthonormal, are often more satisfactory to work with in numerical computation.
Eigenvalues and eigenvectors of an nonsymmetric matrix may be ill-conditioned, that is, sensitive to perturbations in . Chapter f08 contains functions which compute or estimate the condition numbers of eigenvalues and eigenvectors, and the the f08 Chapter Introduction gives more details about the error analysis of nonsymmetric eigenproblems. The accuracy with which eigenvalues and eigenvectors can be obtained is often improved by balancing a matrix. This is discussed further in Section [Balancing for Nonsymmmetric Eigenproblems] below.
Computation of eigenvalues, eigenvectors or the Schur factorization proceeds in the following stages:
All the above remarks also apply – with the obvious changes – to the case when is a complex matrix. The eigenvalues are in general complex, so there is no need for special treatment of complex conjugate pairs, and the Schur form is simply a complex upper triangular matrix.
|
|
|
The Singular Value Decomposition
|
|
The singular value decomposition (SVD) of a real by matrix is given by
where and are orthogonal and is an by diagonal matrix with real diagonal elements, , such that
The are the singular values of and the first columns of and are, respectively, the left and right singular vectors of . The singular values and singular vectors satisfy
where and are the th columns of and respectively.
The singular value decomposition of is closely related to the eigen-decompositions of the symmetric matrices or , because:
However, these relationships are not recommended as a means of computing singular values or vectors.
Singular values are well-conditioned; that is, they are not unduly sensitive to perturbations in . The sensitivity of a singular vector depends on how small the gap is between its singular value and any other singular value: the smaller the gap, the more sensitive the singular vector. More details on the accuracy of computed singular values and vectors are given in the function documents and in the f08 Chapter Introduction.
The singular value decomposition is useful for the numerical determination of the rank of a matrix, and for solving linear least-squares problems, especially when they are rank-deficient (or nearly so). See Chapter f04.
Computation of singular values and vectors proceeds in the following stages:
All the above remarks also apply – with the obvious changes – to the case when is a complex matrix. The singular vectors are complex, but the singular values are real and non-negative, and the bi-diagonal matrix is also real.
|
|
Generalized Eigenvalue Problems
|
|
Let and be square matrices of order . The generalized eigenvalue problem is to find eigenvalues, , and corresponding eigenvectors, , such that
(4)
For given and , the set of all matrices of the form is called a pencil, and and are said to be an eigenvalue and eigenvector of the pencil .
When is non-singular, equation (4) is mathematically equivalent to , and when is non-singular, it is equivalent to . Thus, in theory, if one of the matrices or is known to be nonsingular, the problem could be reduced to a standard eigenvalue problem.
However, for this reduction to be satisfactory from the point of view of numerical stability, it is necessary not only that (or ) should be nonsingular, but that it should be well-conditioned with respect to inversion. The nearer is to singularity, the more unsatisfactory will be as a vehicle for determining the required eigenvalues. Well-determined eigenvalues of the original problem (4) may be poorly determined even by the correctly rounded version of .
We consider first a special class of problems in which is known to be non-singular, and then return to the general case in the following sub-section.
|
Generalized symmetric-definite eigenvalue problems
|
|
If and are symmetric and is positive-definite, then the generalized eigenvalue problem has desirable properties similar to those of the standard symmetric eigenvalue problem. The eigenvalues are all real, and the eigenvectors, while not orthogonal in the usual sense, satisfy the relations for and can be normalized so that .
Note that it is not enough for and to be symmetric; must also be positive-definite, which implies non-singularity. Eigenproblems with these properties are referred to as symmetric-definite problems.
If is the diagonal matrix whose diagonal elements are the eigenvalues, and is the matrix whose columns are the eigenvectors, then
To compute eigenvalues and eigenvectors, the problem can be reduced to a standard symmetric eigenvalue problem, using the Cholesky factorization of as or (see Chapter f07). Note, however, that this reduction does implicitly involve the inversion of , and hence this approach should not be used if is ill-conditioned with respect to inversion.
For example, with , we have
Hence the eigenvalues of are those of , where is the symmetric matrix and . The standard symmetric eigenproblem may be solved by the methods described in Section [Standard symmetric eigenvalue problems]. The eigenvectors of the original problem may be recovered by computing .
Most of the functions which solve this class of problems can also solve the closely related problems
where again and are symmetric and is positive-definite. See the function documents for details.
All the above remarks also apply – with the obvious changes – to the case when and are complex Hermitian matrices. Such problems are called Hermitian-definite. The eigenvectors are complex, but the eigenvalues are all real.
|
|
Generalized nonsymmetric eigenvalue problems
|
|
Any generalized eigenproblem which is not symmetric-definite with well-conditioned must be handled as if it were a general nonsymmetric problem.
If is singular, the problem has infinite eigenvalues. These are not a problem; they are equivalent to zero eigenvalues of the problem . Computationally they appear as very large values.
If and are both singular and have a common null-space, then is singular for all ; in other words, any value can be regarded as an eigenvalue. Pencils with this property are called singular.
As with standard nonsymmetric problems, a real problem may have complex eigenvalues, occurring as complex conjugate pairs.
The generalized eigenvalue problem can be solved via the generalized Schur factorization of and :
where and are orthogonal, is upper triangular, and is upper quasi-triangular (defined just as in Section [Standard nonsymmetric eigenvalue problems]).
If all the eigenvalues are real, then is upper triangular; the eigenvalues are given by . If there are complex conjugate pairs of eigenvalues, then has by diagonal blocks.
Eigenvalues and eigenvectors of a generalized nonsymmetric problem may be ill-conditioned; that is, sensitive to perturbations in or .
Particular care must be taken if, for some , , or in practical terms if and are both small; this means that the pencil is singular, or approximately so. Not only is the particular value undetermined, but also no reliance can be placed on any of the computed eigenvalues. See also the function documents.
Computation of eigenvalues and eigenvectors proceeds in the following stages.
All the above remarks also apply – with the obvious changes – to the case when and are complex matrices. The eigenvalues are in general complex, so there is no need for special treatment of complex conjugate pairs, and the matrix in the generalized Schur factorization is simply a complex upper triangular matrix.
|
|
|
|
Recommendations on Choice and Use of Available Functions
|
|
|
Black Box Functions and General Purpose Functions
|
|
Functions in the NAG Library for solving eigenvalue problems fall into two categories.
|
Black Box Functions: these are designed to solve a standard type of problem in a single call – for example, to compute all the eigenvalues and eigenvectors of a real symmetric matrix. You are recommended to use a black box function if there is one to meet your needs; refer to the decision tree in Section [Black Box Functions] or the index in Section [Index].
|
|
General Purpose Functions: these perform the computational subtasks which make up the separate stages of the overall task, as described in Section [Background to the Problems] – for example, reducing a real symmetric matrix to tri-diagonal form. General purpose functions are to be found, for historical reasons, some in this chapter, a few in Chapter f01, but most in Chapter f08. If there is no black box function that meets your needs, you will need to use one or more general purpose functions.
|
|
Here are some of the more likely reasons why you may need to do this:
|
|
Your problem is already in one of the reduced forms – for example, your symmetric matrix is already tri-diagonal.
|
The decision trees in Section [General Purpose Functions (Eigenvalues and Eigenvectors)] list the combinations of general purpose functions which are needed to solve many common types of problem.
Sometimes a combination of a black box function and one or more general purpose functions will be the most convenient way to solve your problem: the black box function can be used to compute most of the results, and a general purpose function can be used to perform a subsidiary computation, such as computing condition numbers of eigenvalues and eigenvectors.
|
|
Computing Selected Eigenvalues and Eigenvectors
|
|
The decision trees and the function documents make a distinction between functions which compute all eigenvalues or eigenvectors, and functions which compute selected eigenvalues or eigenvectors; the two classes of function use different algorithms.
It is difficult to give clear guidance on which of these two classes of function to use in a particular case, especially with regard to computing eigenvectors. If you only wish to compute a very few eigenvectors, then a function for selected eigenvectors will be more economical, but if you want to compute a substantial subset (an old rule of thumb suggested more than 25%), then it may be more economical to compute all of them. Conversely, if you wish to compute all the eigenvectors of a sufficiently large symmetric tri-diagonal matrix, the function for selected eigenvectors may be faster.
The choice depends on the properties of the matrix and on the computing environment; if it is critical, you should perform your own timing tests.
For nonsymmetric eigenproblems, there are no algorithms provided for computing selected eigenvalues; it is always necessary to compute all the eigenvalues, but you can then select specific eigenvectors for computation by inverse iteration.
|
|
Storage Schemes for Symmetric Matrices
|
|
Functions which handle symmetric matrices are usually designed to use either the upper or lower triangle of the matrix; it is not necessary to store the whole matrix. If either the upper or lower triangle is stored conventionally in the upper or lower triangle of a two-dimensional array, the remaining elements of the array can be used to store other useful data. However, that is not always convenient, and if it is important to economize on storage, the upper or lower triangle can be stored in a one-dimensional array of length ; in other words, the storage is almost halved. This storage format is referred to as packed storage.
Functions designed for packed storage are usually less efficient, especially on high-performance computers, so there is a trade-off between storage and efficiency.
A band matrix is one whose non-zero elements are confined to a relatively small number of sub-diagonals or super-diagonals on either side of the main diagonal. Algorithms can take advantage of bandedness to reduce the amount of work and storage required.
Functions which take advantage of packed storage or bandedness are provided for both standard symmetric eigenproblems and generalized symmetric-definite eigenproblems.
|
|
Balancing for Nonsymmmetric Eigenproblems
|
|
There are two preprocessing steps which one may perform on an nonsymmetric matrix in order to make its eigenproblem easier. Together they are referred to as balancing.
Functions are provided in Chapter f08 for performing either or both of these pre-processing steps, and also for transforming computed eigenvectors or Schur vectors back to those of the original matrix.
Black box functions in this chapter which compute the Schur factorization perform only the permutation step, since diagonal scaling is not in general an orthogonal transformation. The black box functions which compute eigenvectors perform both forms of balancing.
|
|
Non-uniqueness of Eigenvectors and Singular Vectors
|
|
Eigenvectors, as defined by equations (1) or (4), are not uniquely defined. If is an eigenvector, then so is where is any non-zero scalar. Eigenvectors computed by different algorithms, or on different computers, may appear to disagree completely, though in fact they differ only by a scalar factor (which may be complex). These differences should not be significant in any application in which the eigenvectors will be used, but they can arouse uncertainty about the correctness of computed results.
Even if eigenvectors are normalized so that , this is not sufficient to fix them uniquely, since they can still be multiplied by a scalar factor such that . To counteract this inconvenience, most of the functions in this chapter, and in Chapter f08, normalize eigenvectors (and Schur vectors) so that and the component of with largest absolute value is real and positive. (There is still a possible indeterminacy if there are two components of equal largest absolute value – or in practice if they are very close – but this is rare.)
In symmetric problems the computed eigenvalues are sorted into ascending order, but in nonsymmetric problems the order in which the computed eigenvalues are returned is dependent on the detailed working of the algorithm and may be sensitive to rounding errors. The Schur form and Schur vectors depend on the ordering of the eigenvalues and this is another possible cause of non-uniqueness when they are computed. However, it must be stressed again that variations in the results from this cause should not be significant. (Functions in Chapter f08 can be used to transform the Schur form and Schur vectors so that the eigenvalues appear in any given order if this is important.)
In singular value problems, the left and right singular vectors and which correspond to a singular value cannot be normalized independently: if is multiplied by a factor such that , then must also be multiplied by .
Non-uniqueness also occurs among eigenvectors which correspond to a multiple eigenvalue, or among singular vectors which correspond to a multiple singular value. In practice, this is more likely to be apparent as the extreme sensitivity of eigenvectors which correspond to a cluster of close eigenvalues (or of singular vectors which correspond to a cluster of close singular values).
|
|
|
Decision Trees
|
|
|
Black Box Functions
|
|
The decision tree for this section is divided into three sub-trees.
|
Tree 1: Eigenvalues and Eigenvectors of Real Matrices
|
|
Tree 2: Eigenvalues and Eigenvectors of Complex Matrices
|
|
Tree 3: Singular Values and Singular Vectors
|
|
Eigenvalues and Eigenvectors of Real Matrices
|
|
Q:1 Is the eigenproblem ?
Q:2 Are and symmetric with positive-definite and well-conditioned w.r.t inversion?
Q:3 Are eigenvalues only required?
Q:4 The eigenproblem is . Is symmetric?
Q:5 Are eigenvalues only required?
Q:6 Are eigenvalues only required?
Q:7 Is the Schur factorization required?
Q:8 Are all eigenvectors required?
|
|
Eigenvalues and Eigenvectors of Complex Matrices
|
|
Q:1 Is the eigenproblem ?
Q:2 The eigenproblem is . Is Hermitian?
Q:3 Are eigenvalues only required?
Q:4 Are eigenvalues only required?
Q:5 Is the Schur factorization required?
Q:6 Are all eigenvectors required?
|
|
Singular Values and Singular Vectors
|
|
Q:1 Is a complex matrix?
|
|
|
General Purpose Functions (Eigenvalues and Eigenvectors)
|
|
The decision tree for this section is divided into eight sub-trees:
|
Tree 1: Real Symmetric Eigenvalue Problems in the f08 Chapter Introduction
|
|
Tree 2: Real Generalized Symmetric-definite Eigenvalue Problems in the f08 Chapter Introduction
|
|
Tree 3: Real Nonsymmetric Eigenvalue Problems in the f08 Chapter Introduction
|
|
Tree 4: Real Generalized Nonsymmetric Eigenvalue Problems in the f08 Chapter Introduction
|
|
Tree 5: Complex Hermitian Eigenvalue Problems in the f08 Chapter Introduction
|
|
Tree 6: Complex Generalized Hermitian-definite Eigenvalue Problems in the f08 Chapter Introduction
|
|
Tree 7: Complex non-Hermitian Eigenvalue Problems in the f08 Chapter Introduction
|
|
Tree 8: Complex Generalized non-Hermitian Eigenvalue Problems in the f08 Chapter Introduction
|
As it is very unlikely that one of the functions in this section will be called on its own, the other functions required to solve a given problem are listed in the order in which they should be called.
|
|
|
See Also
|
|
Anderson E, Bai Z, Bischof C, Blackford S, Demmel J, Dongarra J J, Du Croz J J, Greenbaum A, Hammarling S, McKenney A and Sorensen D (1999) LAPACK Users' Guide (3rd Edition) SIAM, Philadelphia
Golub G H and Van Loan C F (1996) Matrix Computations (3rd Edition) Johns Hopkins University Press, Baltimore
Parlett B N (1998) The Symmetric Eigenvalue Problem SIAM, Philadelphia
Wilkinson J H and Reinsch C (1971) Handbook for Automatic Computation II, Linear Algebra Springer–Verlag
NAG Toolbox Overview.
NAG Web Site.
|
|