Iterative Methods Based on Soft Thresholding of

Hierarchical Tensors
- BY D. SA I P R A NEETH
Contents:
1. Problem
a. Tensor formats
b. Need for improvement
2. Soft Thresholding
3. Priori Approach
4. Posteriori
a. Algorithm 1
b. Algorithm 2 – Inexact residual Evaluation
5. Numerical Experiments
6. Conclusion

2
 Problem Statement
Low rank Representations
• Canonical Format
• Tucker Format
• Hierarchical Format
• Tensor Train Format

Linear System & Elliptic Problem Hierarchical Tensor Train Format

AU = f
A , F – Low ranks
Richardson Iterative Scheme
Uk+1 = Uk + µ(AUk - f )

Uk+1 rank is higher than A and f

Exponential Increase of rank during
the iteration.

No. of Operations : O(d𝒓𝟒 + d𝒓𝟐 n) 3

 Hard Thresholding Soft Thresholding

h(x) :=(1 – Χ[-α , α ](x)) x s(x) := sgn(x) max {|x|-α , 0}

Rank Truncation Rank Reduction

• Expansive • Soft thresholding replaces entries that have absolute

• Near optimality value below the threshold by zero, but also
• Error Controlled decreases all remaining entries, incurring an
additional error.

Galerkin-type Projection
um(ξ) = arg min ||A(v, ξ) - f || - Residual norm

• Error Controlled
• Non-expansive
• Quasi Optimal reduced basis
• Bounded by Maximum rank of the best H-tensor

4
 A Priori Soft thresholding Parameters

The decay of the singular values of u* is known

Remark 3.5

5
 A Posteriori Soft thresholding Parameters
Algorithm 1 Algorithm 2

• Slighly slower rate of linear convergence

• Step 5 ensures that αk never becomes too
small in relation to the corresponding
current error ||uk- u*|| - Minimization
No a priori information of u* is needed
except the hierarchical tensor representation Inexact Residual Computation
of f and the upper and lower bounds of Eigen
values of A
6
 A Posteriori Soft thresholding Parameters

• A perturbed version of Algorithm 1 where residuals are no

longer evaluated exactly, but only up to a certain relative
error
• Adaptive wavelet scheme, the residual error is controlled
relative to the norm of the computed residual

• rk is adjusted twice in the algorithm : first in line 13 to

ensure the accuracy with respect to ||rk|| , and second time
in line 7 (after incrementing k) to ensure the accuracy with
respect to ||uk- u*||

7
Conclusions :
• Soft Thresholding method guarantees linear convergence to the solution u as well as quasi-
optimality of the tensor ranks of all iterates, and is universal in the sense that no a priori
knowledge on the tensor approximability of u is required.

• If the singular values of u* are known to have exponential type decay, same properties can be
obtained by priori prescribing αk that decrease geometrically.

• The approach can be applied to more problems than the linear elliptic problems discussed in
the paper.

• The approach can be extended to adaptive discretizations

8
Reference
• A. Nouy.
Low-rank tensor methods for model order reduction.
ArXiv e-prints, November 2015.
• Lecture Notes
• Lars Grasedyck & Wolfgang Hackbusch
An Introduction to Hierarchical (H-) Rank and TT-Rank of Tensors with Examples

9
THANK YOU

10