高维多变量回归问题非凸正则化理论与算法研究

High-dimensional multivariate regression problem is one of the hot topics in statistics, optimization, information science and other fields, which has broad applications in many aspects such as pattern recognition, machine learning, and artificial intelligence. This project aims to study several non-convex and non-smooth matrix optimization models regularized by (group) sparsity and low rank from high-dimensional multivariate regression problems. We hope to realize the organic combination of their optimization theory, statistical properties, and optimization algorithms. The main research contents are as follows : (1) We will study the properties of the objective functions, such as semi-smoothness, first-order and second-order directional derivatives, and sub-derivative and second sub-derivatives; (2) We will research the optimization theory of the models. Here we will mainly consider several special kinds of stationary points such as the second-order directional stationary point and the second-order stationary point in subspace. We will focus on their properties such as variational property, equivalent descriptions, optimality, sparsity, low rank, and lower bound; (3) We will study the statistical properties of the above stationary points (i.e., estimators) such as asymptotic distribution, unbiasedness, Oracle property, and error bound; (4) We will provide some efficient algorithms for computing the above special stationary points, and analyze their convergences; (5) We will carry out some numerical experiments for testing the models and the algorithms, and apply them to practical problems such as multi-image face recognition and gene expression analysis. Therefore, this project is an advanced subject of basic theory and application research, and so it is of important scientific significance and practical value.

高维多变量回归问题是统计学、最优化、信息科学等领域共同关注的热点课题之一，在模式识别、机器学习与人工智能等方面有广泛应用。本项目研究由高维多变量回归问题之稀疏和低秩正则化得到的几个非凸非光滑矩阵优化模型，希望实现优化理论、统计性质、优化算法的有机结合，主要内容有：（1）研究目标函数的性质，如半光滑性质，一阶方向导数或次导数、二阶方向导数或二次次导数等；（2）研究模型的优化理论，主要考虑几类特殊稳定点（如二阶方向稳定点、子空间二阶稳定点等）的变分性质、等价刻画、最优性、稀疏性、低秩性、下界性质等；（3）研究上述稳定点的统计性质，如渐进分布、无偏性、Oracle性质、误差界等；（4）研究上述特殊稳定点的有效算法，分析算法的收敛性；（5）对模型和算法进行数值试验，并将其用于多图像人脸识别、基因表达分析等实际问题。本项目是一个基础理论与应用研究的前沿课题，具有重要的科学意义和实用价值。

高维多变量回归问题是统计学、最优化、信息科学等领域共同关注的热点课题之一，其稀疏和低秩正则化所诱导的非凸、非光滑、非Lipschitz的优化问题给最优化理论和方法带来了诸多挑战和机遇。本项目对几类相关问题建立了最优性理论和相关性质，提供了有效算法，主要成果有：（1）对一类组稀疏约束优化问题，建立了其局部最优解、四类稳定点之间的关系，给出了组稀疏约束优化问题一阶和二阶最优性条件，提出了改进的迭代硬阈值算法，分析算法的收敛性质，验证了算法的有效性。（2）对一类带复合非凸惩罚的组稀疏优化模型，研究了模型的一阶和二阶最优性条件及一阶二阶方向稳定点的性质，提出了光滑化信赖域牛顿算法，数值算例验证了理论和方法的正确性。（3）对一类损失函数为非光滑凸函数且带有约束的高维组稀疏问题研究其精确连续松弛，给出了三类稳定点的特征刻画及其关系，讨论了松弛问题和原问题解的等价性，提出了组光滑化邻近梯度算法，证明了该算法全局收敛性。（4）研究了一类带lp正则的最小二乘回归极小化问题，提出光滑化拟牛顿算法，通过同步更新光滑化参数和正则化参数，实现了参数的自适应调整。（5）对损失为一般光滑函数带有稀疏加组稀疏正则项的高维回归问题,分析了方向稳定点的特征和最优性，构造了光滑化逼近问题，证明了稳定点的一致性。（6）考虑了一类基于Huber损失和线性不等式约束的稀疏优化模型，提出了光滑化惩罚算法, 并证明了该算法的收敛性。（7）研究了一类带lp（0<p<1）正则项的一般矩阵优化问题，提出了奇异半阈值算法，证明了算法的收敛性。（8）对一类低秩稀疏分解问题提出双非凸松弛模型和参数连续化增广拉格朗日交替方向法，证明了该算法的收敛性。（9）对几类最优化相关问题（向量平衡、微分博弈、群体博弈等）研究了解的稳定性和逼近问题。.本研究丰富和发展了非光滑、非凸、非Lipschitz优化的最优性理论和有效算法。