This class implements a Gaussian Process (GP) regression model for non-parametric Bayesian regression.
More...
|
| | GaussianProcess () |
| |
| | ~GaussianProcess ()=default |
| |
| | GaussianProcess (const std::shared_ptr< covariance::Covariance > &kernel, const std::shared_ptr< mean::Mean > &mean, Eigen::Ref< const Eigen::VectorXd > params, double nugget=1e-8) |
| |
| | GaussianProcess (const GaussianProcess &other) |
| |
| GaussianProcess & | operator= (const GaussianProcess &other) |
| |
| | GaussianProcess (GaussianProcess &&other) noexcept |
| |
| GaussianProcess & | operator= (GaussianProcess &&other) noexcept |
| |
| void | set (const std::shared_ptr< covariance::Covariance > &kernel, const std::shared_ptr< mean::Mean > &mean, Eigen::Ref< const Eigen::VectorXd > params, double nugget=1e-8) |
| |
| void | condition (const Eigen::Ref< const Eigen::MatrixXd > &xObs, const Eigen::Ref< const Eigen::VectorXd > &yObs, bool copyData=true, bool normalizeY=false) |
| | Condition the GP on a set of observations (allow predictive posterior computation)
|
| |
| void | fit (const Eigen::Ref< const Eigen::MatrixXd > &xObs, const Eigen::Ref< const Eigen::VectorXd > &yObs, const Eigen::Ref< const Eigen::VectorXd > &lb, const Eigen::Ref< const Eigen::VectorXd > &ub, const method &method, const nlopt::algorithm &alg, const double &tol_rel, bool copyData=true, bool normalizeY=false, const std::shared_ptr< cmp::prior::Prior > &prior=cmp::prior::Uniform::make(), const std::vector< bool > &logScale={}) |
| | Fit the Gaussian Process to the observations.
|
| |
| Eigen::VectorXd | getParameters () const |
| | Get the hyperparameters of the Gaussian Process.
|
| |
| std::shared_ptr< cmp::mean::Mean > | getMean () const |
| | Gets the shared pointer to the GP prior mean function.
|
| |
| std::shared_ptr< cmp::covariance::Covariance > | getKernel () const |
| | Gets the shared pointer to the GP covariance kernel.
|
| |
| double | getNugget () const |
| | Gets the observation noise variance (nugget).
|
| |
| Eigen::MatrixXd | covariance (Eigen::Ref< const Eigen::VectorXd > par) const |
| | Evaluates the covariance matrix of the Kernel.
|
| |
| Eigen::MatrixXd | covarianceGradient (Eigen::Ref< const Eigen::VectorXd > par, const int &i) const |
| |
| Eigen::MatrixXd | covarianceHessian (Eigen::Ref< const Eigen::VectorXd > par, const size_t &i, const size_t &j) const |
| |
| const Eigen::LDLT< Eigen::MatrixXd > & | getCovDecomposition () const |
| |
| const Eigen::VectorXd & | getAlpha () const |
| |
| const Eigen::VectorXd & | getDiagCovInverse () const |
| |
| const Eigen::VectorXd & | getResidualVector () const |
| |
| const Eigen::Ref< const Eigen::MatrixXd > & | getXObs () const |
| |
| const Eigen::Ref< const Eigen::VectorXd > & | getYObs () const |
| |
| size_t | nObs () const |
| | Returns the number of training observations.
|
| |
| Eigen::VectorXd | priorMean (Eigen::Ref< const Eigen::VectorXd > par) const |
| | Evaluates the mean on the observations.
|
| |
| Eigen::VectorXd | priorMeanGradient (Eigen::Ref< const Eigen::VectorXd > par, const int &i) const |
| | Evaluate the gradient of the mean function.
|
| |
| Eigen::VectorXd | residual (Eigen::Ref< const Eigen::VectorXd > par) const |
| | Evaluate the difference between the observation and the mean function.
|
| |
| std::pair< double, double > | predict (const Eigen::Ref< const Eigen::VectorXd > &x, type predictionType=type::POSTERIOR) const |
| | Compute the predictive mean and variance at a new point.
|
| |
| double | predictMean (const Eigen::Ref< const Eigen::VectorXd > &x, type predictionType=type::POSTERIOR) const |
| | Compute the predictive mean at a new point.
|
| |
| std::pair< Eigen::VectorXd, Eigen::MatrixXd > | predictMultiple (const Eigen::Ref< const Eigen::MatrixXd > &x_pts, type predictionType=type::POSTERIOR) const |
| | Compute the predictive variance at a new set of points.
|
| |
| Eigen::VectorXd | predictMeanMultiple (const Eigen::Ref< const Eigen::MatrixXd > &x_pts, type predictionType=type::POSTERIOR) const |
| | Compute the predictive mean at a new set of points.
|
| |
| std::pair< double, double > | predictLOO (const size_t &i) const |
| | Compute the leave-one-out predictive mean and variance at the i-th observation point.
|
| |
| double | logLikelihood () const |
| | Compute the log-likelihood of the observations given the hyperparameters.
|
| |
| double | logLikelihoodLOO (const size_t &i) const |
| | Compute the leave-one-out log-likelihood of the observations given the hyperparameters.
|
| |
| Eigen::MatrixXd | expectedVarianceImprovement (const Eigen::Ref< const Eigen::MatrixXd > &x_pts, const Eigen::Ref< const Eigen::MatrixXd > &x_pending, double nu) const |
| |
| double | objectiveFunction (const Eigen::Ref< const Eigen::VectorXd > &x, Eigen::Ref< Eigen::VectorXd > grad, const std::shared_ptr< cmp::prior::Prior > &prior) |
| |
| double | objectiveFunctionLOO (const Eigen::Ref< const Eigen::VectorXd > &x, Eigen::Ref< Eigen::VectorXd > grad, const std::shared_ptr< cmp::prior::Prior > &prior) |
| |
| double | objectiveFunctionLOOMSE (const Eigen::Ref< const Eigen::VectorXd > &x, Eigen::Ref< Eigen::VectorXd > grad, const std::shared_ptr< cmp::prior::Prior > &prior) |
| |
|
| Eigen::VectorXd | par_ |
| | Model hyperparameters.
|
| |
| std::shared_ptr< covariance::Covariance > | pKernel_ |
| | Covariance kernel function.
|
| |
| std::shared_ptr< mean::Mean > | pMean_ |
| | Prior mean function.
|
| |
| double | nugget_ |
| | Observation noise variance (nugget).
|
| |
| std::optional< Eigen::MatrixXd > | xObs_ |
| | Owning storage for training input matrix.
|
| |
| std::optional< Eigen::VectorXd > | yObs_ |
| | Owning storage for training target vector.
|
| |
| std::optional< Eigen::Ref< const Eigen::MatrixXd > > | pXObs_ |
| | Reference wrapper to training input matrix.
|
| |
| std::optional< Eigen::Ref< const Eigen::VectorXd > > | pYObs_ |
| | Reference wrapper to training target vector.
|
| |
| Eigen::LDLT< Eigen::MatrixXd > | covDecomposition_ |
| | LDLT decomposition of the training covariance matrix.
|
| |
| Eigen::VectorXd | alpha_ |
| | GP weights vector: alpha = (K + s^2 I)^-1 (y - m).
|
| |
| Eigen::VectorXd | diagCovInverse_ |
| | Diagonal elements of the inverse covariance matrix.
|
| |
| Eigen::VectorXd | residual_ |
| | Residuals of the mean function: y - m(x).
|
| |
| bool | normalizeY_ = false |
| | Flag indicating whether target normalization is active.
|
| |
| cmp::scaler::StandardScaler | yScaler_ |
| | Scaler used to normalize the target vector.
|
| |
This class implements a Gaussian Process (GP) regression model for non-parametric Bayesian regression.
Mathematical Foundations
A Gaussian Process is a collection of random variables, any finite number of which have a joint Gaussian distribution. It is completely specified by its mean function \(m(\mathbf{x})\) and covariance (kernel) function \(k(\mathbf{x}, \mathbf{x}')\):
\[ f(\mathbf{x}) \sim \mathcal{GP}\left(m(\mathbf{x}), k(\mathbf{x}, \mathbf{x}')\right) \]
Given a training dataset \(\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^N\) where \(\mathbf{X} \in \mathbb{R}^{N \times D}\) and \(\mathbf{y} \in \mathbb{R}^N\), the joint distribution of the training outputs and the predictive function value \(f(\mathbf{x}_*)\) at a test point \(\mathbf{x}_*\) is:
\[
\begin{bmatrix} \mathbf{y} \\ f(\mathbf{x}_*) \end{bmatrix} \sim \mathcal{N}\left(
\begin{bmatrix} \mathbf{m}(\mathbf{X}) \\ m(\mathbf{x}_*) \end{bmatrix},
\begin{bmatrix} \mathbf{K}(\mathbf{X}, \mathbf{X}) + \sigma_n^2 \mathbf{I} & \mathbf{k}(\mathbf{X}, \mathbf{x}_*) \\ \mathbf{k}(\mathbf{x}_*, \mathbf{X}) & k(\mathbf{x}_*, \mathbf{x}_*) \end{bmatrix}
\right)
\]
where \(\sigma_n^2\) is the nugget (representing the observation noise variance).
The conditional posterior distribution at the test point \(\mathbf{x}_*\) is given by:
\[ f(\mathbf{x}_*) | \mathbf{X}, \mathbf{y}, \mathbf{x}_* \sim \mathcal{N}(\mu_*, \sigma_*^2) \]
where the posterior mean \(\mu_*\) and variance \(\sigma_*^2\) are calculated as:
\[ \mu_* = m(\mathbf{x}_*) + \mathbf{k}(\mathbf{x}_*, \mathbf{X}) \left[ \mathbf{K}(\mathbf{X}, \mathbf{X}) + \sigma_n^2 \mathbf{I} \right]^{-1} (\mathbf{y} - \mathbf{m}(\mathbf{X})) \]
\[ \sigma_*^2 = k(\mathbf{x}_*, \mathbf{x}_*) - \mathbf{k}(\mathbf{x}_*, \mathbf{X}) \left[ \mathbf{K}(\mathbf{X}, \mathbf{X}) + \sigma_n^2 \mathbf{I} \right]^{-1} \mathbf{k}(\mathbf{X}, \mathbf{x}_*) \]
Implementation Algorithms
- LDLT Decomposition: Directly inverting \(\mathbf{K}_y = \mathbf{K}(\mathbf{X}, \mathbf{X}) + \sigma_n^2 \mathbf{I}\) is numerically unstable and computationally expensive ( \(O(N^3)\)). We construct \(\mathbf{K}_y\) and compute its LDLT decomposition:
\[ \mathbf{K}_y = \mathbf{L} \mathbf{D} \mathbf{L}^T \]
- Backsubstitution: We precompute the weight vector \(\boldsymbol{\alpha}\) by solving:
\[ \mathbf{L} \mathbf{D} \mathbf{L}^T \boldsymbol{\alpha} = \mathbf{y} - \mathbf{m}(\mathbf{X}) \]
The predictive posterior mean is then efficiently evaluated as:
\[ \mu_* = m(\mathbf{x}_*) + \mathbf{k}(\mathbf{x}_*, \mathbf{X}) \boldsymbol{\alpha} \]
- Hyperparameter Optimization: The hyperparameters \(\boldsymbol{\theta}\) of the covariance and mean functions are optimized by maximizing the marginal log-likelihood:
\[ \log p(\mathbf{y} | \mathbf{X}, \boldsymbol{\theta}) = -\frac{1}{2} (\mathbf{y} - \mathbf{m}(\mathbf{X}))^T \mathbf{K}_y^{-1} (\mathbf{y} - \mathbf{m}(\mathbf{X})) - \frac{1}{2} \log |\mathbf{K}_y| - \frac{N}{2} \log (2\pi) \]
using gradient-free (e.g., Subplex) or gradient-based NLopt algorithms.
Constraints & Invariants
- **Nugget ( \(\sigma_n^2\))**: Must be strictly positive ( \(\ge 10^{-12}\), default \(10^{-8}\)) to ensure the covariance matrix remains strictly positive-definite.
- Training Inputs: Pre-conditions require that the dataset is non-empty ( \(N \ge 1\)).
- Hyperparameter bounds: Must satisfy the user-specified bounds \(\mathbf{lb} \le \boldsymbol{\theta} \le \mathbf{ub}\).