Title: ETH S22 Empirical Process Theory and Applications
Tags: lecture-02 lecture-08 lecture-05 lecture-06 lecture-03 lecture-09 lecture-07 lecture-04 lecture-11 lecture-10 lecture-12

Front: [latex]
Suppose $X_1, \dots, X_n, \dots $ are i.i.d. random variables sampled according to some distribution $P$. What does $\hat{P}_n $ mean in this case?
[/latex]
Back: [latex]
$\hat{P}_n $ is the empirical measure, defined for each measurable set $A$ as
\[
\hat{P}_n = \frac{\text{number of $X_i \in A$} }{n} = \frac{1}{n} \#\left\{ X_i \in A, 1 \le i \le n \right\}.
\]
[/latex]

Front: [latex]
Suppose $X_1, \dots, X_n, \dots $ are i.i.d. random variables sampled according to some distribution $P$. How do you define the empirical distribution function $\hat{F}_n $?
[/latex]
Back: [latex]
\[
\hat{F}_n (x) = \frac{\text{number of $X_i \le x$} }{n} = \frac{1}{n}\# \left\{ X_i \le x, 1 \le i \le n \right\}.
\]
[/latex]

Front: [latex]
Consider a function $g : \X \to \R$, a probability measure $P$, and a corresponding empirical measure $\hat{P}_n $. What do these mean?
\begin{enumerate}[(a)]
\item $Pg$
\item $\hat{P}_n g$
\end{enumerate}
[/latex]
Back: [latex]
\begin{enumerate}[(a)]
\item $Pg := \Exnb g(X) = \int_{}^{} gdP$
\item $\hat{P}_n g := \frac{1}{n} \sum_{i = 1}^{n} g(X_i) = \int_{}^{} g d\hat{P}_n $
\end{enumerate}
[/latex]

Front: [latex]
What does it mean for a sequence of random variables $Z_n$ to converge to $Z$ in probability?
[/latex]
Back: [latex]
$Z_n \xrightarrow[n \to \infty]{\P} Z$ (converges in probability) if for any $\varepsilon > 0$,
\[
\lim_{n \to \infty} \P \left( \left| Z_n - Z \right| > \varepsilon \right) = 0.
\]
[/latex]

Front: [latex]
What does it mean for a sequence of random variables $Z_n$ to converge to $Z$ almost surely?
[/latex]
Back: [latex]
$Z_n \xrightarrow[n \to \infty]{\text{a.s.} } Z$ (converges almost surely) if
\[
\P\left( \lim_{n \to \infty} Z_n = Z \right) = 1.
\]
[/latex]

Front: [latex]
What is a Glivenko-Cantelli (GC) class of sets?
[/latex]
Back: [latex]
Let $\D$ be a class of measurable subsets of the observation space $\X$. $\D$ is called a Glivenko-Cantelli (GC) class if
\[
\sup_{D \in \D} \left| \hat{P}_n (D) - P(D) \right| \xrightarrow[n \to \infty]{\P} 0,
\] i.e. the uniform law of large numbers (ULLN) holds over $\D$.
[/latex]

Front: [latex]
Let $\X = \R$. Why is the family of half-intervals $\D = \left\{ \left( -\infty, x \right] : x \in \R \right\}$ a Glivenko-Cantelli class?
[/latex]
Back: [latex]
If $\D = \left\{ \left( -\infty, x \right] : x \in \R \right\}$, then
\[
\sup_{D \in \D} \left| P_n(D) - P(D) \right| = \sup_x \left| \hat{F}_n (x) - F(x) \right| \xrightarrow[n \to \infty]{\P} 0
\] by the classical Glivenko-Cantelli theorem.
[/latex]

Front: [latex]
Let $\X = \R$, $\P$ the probability measure of a continuous distribution, and $\D$ the family of all finite subsets of $\R$. Is $\D$ a Glivenko-Cantelli class? Justify your answer.
[/latex]
Back: [latex]
$\D$ is not GC.
For all $D \in \D$, $P(D) = 0$, but $\hat{P}_n \left( \left\{ x_1, \dots, x_n \right\} \right) = 1$. So,
\[
\sup _{D \in \D} \left| \hat{P}_n (D) - P(D) \right| = 1
\] which does not converge to 0, showing that $\D$ is not GC.
[/latex]

Front: [latex]
Suppose $\G$ is a family of functions, $P$ is a probability measure, and $\hat{P}_n $ is a corresponding empirical measure. What does $\left\Vert \hat{P}_n - P \right\Vert _\G$ mean?
[/latex]
Back: [latex]
\[
\left\Vert \hat{P}_n - P \right\Vert := \sup _{g \in \G} \left| \left( \hat{P}_n - P \right)g \right|
\]
[/latex]

Front: [latex]
What is a Glivenko-Cantelli (GC) class of functions?
[/latex]
Back: [latex]
Let $\G$ be a class of (measurable) functions $g : \X \to \R$. We call $\G$ Glivenko-Cantelli if
\[
\left\Vert \hat{P}_n - P \right\Vert _{\G} := \sup _{g \in \G} \left| \left( \hat{P}_n - P \right)g \right| \xrightarrow[n \to \infty]{\P} 0,
\] i.e. the uniform law of large numbers (ULLN) holds over $\G$.
[/latex]

Front: [latex]
Suppose the unknown distribution $P$ comes from a family parametrized by $\theta $, and suppose also that the densities $p_\theta := \frac{\partial P_\theta }{d \mu}$ exist. What is the MLE $\hat{\theta }_n $ of $\theta $? What is $\hat{p}_n $?
[/latex]
Back: [latex]
The MLE $\hat{\theta }_n $ is
\[
\hat{\theta }_n = \arg \max_{\theta \in \Theta} \hat{P}_n \log p_\theta .
\]
$\hat{p}_n $ is the estimated density, defined as
\[
\hat{p}_n := p_{\hat{\theta }_n } = \arg \max_{p \in \Pc} \hat{P}_n \log p
\] where $\Pc$ is the collection of densities in our model.
[/latex]

Front: [latex]
What is the Hellinger distance between densities $p$ and $p_0$?
[/latex]
Back: [latex]
\[
h(p, p_0) := \left( \frac{1}{2}\int_{}^{} \left( \sqrt{p} - \sqrt{p_0} \right)^{2}d\mu \right)^{\frac{1}{2}}
\]
[/latex]

Front: [latex]
Suppose $P$ is some probability measure. Let $p_0$ be its true density, and $\hat{p}_n $ the density obtained from MLE. Show that
\[
\frac{1}{2} P \log \frac{\hat{p}_n }{p_0} \le -h^{2}\left( \hat{p}_n ,p_0 \right)
\] where $h$ is the Hellinger distance defined as
\[
h(p, p_0) := \left( \frac{1}{2}\int_{}^{} \left( \sqrt{p} - \sqrt{p_0} \right)^{2}d\mu \right)^{\frac{1}{2}}.
\]
[/latex]
Back: [latex]
\[\begin{aligned}
\frac{1}{2} P \log \frac{\hat{p}_n }{p_0} &= P \log \sqrt{\frac{\hat{p}_n }{p_0}} &\left( \frac{1}{a}\log x = \log x^{\frac{1}{a}} \right) \\
&\le P \sqrt{\frac{\hat{p}_n }{p_0}} - 1 &\left( \log x \le x - 1 \right) \\
&= \int_{}^{} \sqrt{ \frac{\hat{p}_n }{p_0}} p_0d\mu - 1 \\
&= \int_{}^{} \sqrt{\hat{p}_n } \sqrt{p_0}d\mu - 1\\
&= - \frac{1}{2} \left( \int_{}^{} \hat{p}_n d\mu + \int_{}^{} p_0d\mu - 2\int_{}^{} \sqrt{\hat{p}_n } \sqrt{p_0}d\mu \right) & \left( \int_{}^{} p d\mu = 1 \right) \\
&= - h^{2}(\hat{p}_n , p_0).
\end{aligned}\]
[/latex]

Front: [latex]
Define $\G := \left\{ \frac{1}{2}\log \frac{p}{p_0} : p \in \Pc \right\}$, where $\Pc = \left\{ p_\theta : \theta \in \Theta \right\}$. Suppose $\G$ is GC. Show that then MLE is consistent in the Hellinger distance, i.e. that
\[
h(\hat{p}_n , p_\theta ) \xrightarrow[n \to \infty]{\P} 0.
\]
Recall that the Hellinger distance is defined as
\[
h(p, p_0) := \left( \frac{1}{2}\int_{}^{} \left( \sqrt{p} - \sqrt{p_0} \right)^{2}d\mu \right)^{\frac{1}{2}}.
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item We rely on non-negativity of $\hat{P}_n \log \frac{\hat{p}_n }{p_0}$, and split it into two parts. One of them goes to 0, while the other we can bound with the Hellinger distance.
\[\begin{aligned}
0 &\le \frac{1}{2}\hat{P}_n \log \frac{\hat{p}_n }{p_0} & (\hat{p}_n =\arg \max_{p \in \Pc} \hat{P}_n \log p) \\
&= \underbrace{\frac{1}{2}\left( \hat{P}_n - P \right)\log \frac{\hat{p}_n }{p_0}}_{\xrightarrow[n \to \infty]{\P} 0 \text{ since $\G$ is GC} } + \frac{1}{2}P \log \frac{\hat{p}_n }{p_0}
\end{aligned}\]
\item We use properties of the $\log $ and some juggling to bound \[\frac{1}{2}P \log \frac{\hat{p}_n }{p_0} \le -h^{2}\left( \hat{p}_n , p_0 \right)\] as follows
\[\begin{aligned}
\frac{1}{2} P \log \frac{\hat{p}_n }{p_0} &= P \log \sqrt{\frac{\hat{p}_n }{p_0}} &\left( \frac{1}{a}\log x = \log x^{\frac{1}{a}} \right) \\
&\le P \sqrt{\frac{\hat{p}_n }{p_0}} - 1 &\left( \log x \le x - 1 \right) \\
&= - \frac{1}{2} \left( \int_{}^{} \hat{p}_n d\mu + \int_{}^{} p_0d\mu - 2\int_{}^{} \sqrt{\hat{p}_n } \sqrt{p_0}d\mu \right) & \left( \int_{}^{} p d\mu = 1 \right) \\
&= - h^{2}(\hat{p}_n , p_0).
\end{aligned}\]
\item We can plug 2. into 1. to put an upper bound on $h^{2}\left( \hat{p}_n , p_0 \right)$
\[\begin{aligned}
0 &\le \frac{1}{2}\left( \hat{P}_n - P \right)\log \frac{\hat{p}_n }{p_0} + \frac{1}{2}P \log \frac{\hat{p}_n }{p_0}\\
&\le \frac{1}{2}\left( \hat{P}_n - P \right)\log \frac{\hat{p}_n }{p_0} - h^{2}\left( \hat{p}_n ,p_0 \right) \\
h^{2}\left( \hat{p}_n , p_0 \right) &\le \frac{1}{2}\left( \hat{P}_n - P \right)\log \frac{\hat{p}_n }{p_0} \\
&\xrightarrow[n \to \infty]{\P} 0.
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
State theorem about consistency of MLE.
[/latex]
Back: [latex]
$\G := \left\{ \frac{1}{2}\log \frac{p}{p_0} : p \in \Pc \right\}$, where $\Pc = \left\{ p_\theta : \theta \in \Theta \right\}$. Suppose $\G$ is GC. Then MLE is consistent in the Hellinger distance, i.e. then
\[
h(\hat{p}_n , p_\theta ) \xrightarrow[n \to \infty]{\P} 0.
\]
[/latex]

Front: [latex]
Let $X \in \R$ and $\phi : \R \to \R_+$ increasing. Prove Chebyshev's inequality, i.e. show that then for all $a$
\[
P\left( X \ge a \right) \le \frac{E \phi(X)}{\phi \left( a \right)}.
\]
[/latex]
Back: [latex]
\[\begin{aligned}
E \phi(X) &= \int_{}^{} \phi(x)dP(x) \\
&\ge \int_{x \ge a}^{} \phi(x) dP(x) \\
&\ge \phi(a) \int_{x \ge a}^{} dP(x)\\
&= \phi(x) P(X \ge a).
\end{aligned}\]
[/latex]

Front: [latex]
State Chebyshev's inequality.
[/latex]
Back: [latex]
Let $X \in \R$ and $\phi : \R \to \R_+$ increasing. Then for all $a$
\[
P\left( X \ge a \right) \le \frac{E \phi(X)}{\phi \left( a \right)}.
\]
[/latex]

Front: [latex]
Suppose $X$ is a random variable in $\R$, with $EX := \mu$, $\var{X} = \sigma ^{2} < \infty$. Let $X_1 , \dots , X_n, \dots$ i.i.d. $ \sim X$, $\overline{X}_n = \frac{1}{n} \sum_{i = 1}^{n} X_i$. Show that then
\[\begin{aligned}
\overline{X}_n \xrightarrow[n \to \infty]{\P} \mu.
\end{aligned}\]
[/latex]
Back: [latex]
We can use Chebyshev and finiteness of variance to show
\[\begin{aligned}
\P \left( \left| \overline{X}_n - \mu \right| \ge a \right) &\underbrace{\le}_{\phi(\cdot) = \left( \cdot \right)^{2}1\left( \cdot > 0 \right)} \frac{E\left( \overline{X}_n - \mu \right)^{2}}{a^{2}}\\
&=\frac{\sigma ^{2}}{na^{2}} \xrightarrow[n \to \infty]{} 0.
\end{aligned}\]
Then
\[\begin{aligned}
\overline{X}_n \xrightarrow[n \to \infty]{\P} \mu.
\end{aligned}\]
This is the weak law of large numbers.
[/latex]

Front: [latex]
What does $Z_n = \O_\P (1)$ mean for a sequence of random variables $\left( Z_{n} \right)_{n \geq 1} $? What would $\O_\P \left( 1 / \sqrt{n} \right)$ mean?
[/latex]
Back: [latex]
$Z_n = \O_\P (1)$ means $Z_n$ is bounded in probability, so
\[
\lim_{M \to \infty} \lim_{n \to \infty} \P \left( \left| Z_n \right| > M \right) = 0.
\]
$Z_n = \O_\P\left( 1/\sqrt{n} \right)$ means $\sqrt{n} \cdot Z_n = \O_\P\left( 1 \right)$.
[/latex]

Front: [latex]
Suppose $X$ is a random variable in $\R$, with $EX := \mu$, $\var{X} = \sigma ^{2} < \infty$. Let $X_1 , \dots , X_n, \dots$ i.i.d. $ \sim X$, $\overline{X}_n = \frac{1}{n} \sum_{i = 1}^{n} X_i$. Show that then $\overline{X}_n = \mu + \O_\P\left( 1/\sqrt{n} \right)$.
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item From the weak law of large numbers, we know that
\[
\overline{X}_n \xrightarrow[n \to \infty]{\P} \mu.
\]
\item Now note that for all $t$
\[\begin{aligned}
\P \left( \left| \overline{X}_n - \mu \right| \ge \sigma \sqrt{\frac{t}{n}} \right) &\le \frac{\Exnb \left( \overline{X}_n - \mu \right)^{2}}{\sigma ^{2}t/n} &\text{(Chebyshev)}\\
&= \frac{\sigma ^{2}/n}{\sigma ^{2}t/n} = \frac{1}{t} \\
&\xrightarrow[t \to \infty]{} 0,
\end{aligned}\]
so $\sqrt{n} \left( \overline{X}_n - \mu \right)$ is bounded in probability, i.e. fulfils
\[
\lim_{M \to \infty} \lim_{n \to \infty} \P \left(\sqrt{n} \left| \overline{X}_n - \mu \right| > M \right) = 0.
\]
\item Therefore, we can write $\overline{X}_n = \mu + \O_\P\left( 1/\sqrt{n} \right)$.
\end{enumerate}
[/latex]

Front: [latex]
Consider $X_1, \dots, X_n$ i.i.d. $\sim X \in \R$, and some function $g : \R \to \R$. How can you write $\var{g(X)} $ using the empirical process $\hat{P}_n $ and $P$ notation?
[/latex]
Back: [latex]
\[
\var{g(X)} = n\cdot\Exnb \left[ \left( \hat{P}_n - P \right)g \right]^{2}
\]
since
\[\begin{aligned}
\Exnb \left[ \left( \hat{P}_n - P \right)g \right]^{2} &= \Exnb \left[ \frac{1}{n} \sum_{i = 1}^{n} \left( g\left( X_i \right) - \Exnb g\left( X_i \right) \right) \right] ^{2}\\
&=\frac{1}{n^{2}}\sum_{i = 1}^{n} \Exnb \left[ g\left( X_i \right) - \Exnb g\left( X_i \right) \right]^{2}\\
&=\frac{1}{n} \var{g\left( X \right)}.
\end{aligned}\]
[/latex]

Front: [latex]
Let $\G_N = \left\{ g_1, \dots, g_N \right\}$, $X_1 , \dots , X_n \dots$ i.i.d. $\sim X$. Suppose $\var{g_j(X)}\le \sigma ^{2} < \infty$. Show that $\left\Vert \hat{P}_n - P \right\Vert_{\G_N} = \O_\P\left( \sqrt{\frac{N}{n}} \right)$.
[/latex]
Back: [latex]
First note that
\[\begin{aligned}
\P \left( \left\Vert \hat{P}_n - P \right\Vert _{\G_N} \ge a \right) &= \P\left( \max_{1 \le j \le N} \left| \left( \hat{P}_n - P \right)g_j \right| \ge a \right) &\text{(Def. of $\left\Vert \cdot \right\Vert _{\G_N}$)} \\
&\le \sum_{j = 1}^{N} \max_{1 \le j \le n} \P\left( \left| \left( \hat{P}_n - P \right)g_j \right| \ge a \right) &\text{(Union bound)} \\
&\le \sum_{j = 1}^{N} \max_{1 \le j \le n} \frac{\Exnb \left[ \left( \hat{P}_n - P \right)g_j \right]^{2}}{a^{2}} &\text{(Chebyshev with $u \mapsto u^{2}$)} \\
&\le N \frac{\sigma ^{2}}{na^{2}}.
\end{aligned}\]
Now reformulate the above by setting $a = \sqrt{\frac{Nt}{n}} $ to get
\[\begin{aligned}
\P \left( \left\Vert \hat{P}_n - P \right\Vert _{\G_N} \ge \sqrt{\frac{Nt}{n}} \right) \le \frac{1}{t}.
\end{aligned}\] So $\left\Vert \hat{P}_n - P \right\Vert _{\G_N} = \O_\P \left( \sqrt{\frac{N}{n}} \right)$ for $\sigma $ fixed.
[/latex]

Front: [latex]
Let $X$ be $\No\left( 0, 1 \right)$-distributed. Show that for $\lambda > 0$,
\[
\Exnb \exp \left[ \lambda X \right] = \exp \left[ \lambda ^{2}/2 \right].
\]
[/latex]
Back: [latex]
\[\begin{aligned}
\Exnb \exp \left[ \lambda X \right] &= \int_{}^{} \exp \left[ \lambda x \right] \frac{1}{\sqrt{2\pi} } \exp \left[ -\frac{1}{2}x^{2} \right] dx \\
&=\exp \left[ \lambda ^{2}/2 \right]\underbrace{\int_{}^{} \frac{1}{\sqrt{2 \pi} } \exp \left[ - \frac{1}{2}\left( x - \lambda \right)^{2} \right] dx}_{=1 \text{ because pdf of $\No\left( \lambda , 1 \right)$} } \\
&= \exp \left[ \lambda ^{2}/2 \right].
\end{aligned}\]
[/latex]

Front: [latex]
Let $X$ be $\No\left( 0, 1 \right)$-distributed. Assume that for $\lambda > 0$,
\[
\Exnb \exp \left[ \lambda X \right] = \exp \left[ \lambda ^{2}/2 \right].
\] Show that for all $a > 0$,
\[
P\left( X \ge a \right) \le \exp \left[ \lambda ^{2} / 2 - \lambda a \right]
\] and conclude, by taking $\lambda =a$, that
\[
P\left( X \ge a \right) \le \exp \left[ -a^{2}/2 \right].
\]
[/latex]
Back: [latex]
We can simply apply Chebyshev
\[\begin{aligned}
P(X \ge a) &\le \frac{\Exnb \exp \left[ \lambda X \right]}{\exp [\lambda a]} &\text{(Chebyshev)} \\
&=\exp \left[ \lambda ^{2}/2 \right] \exp \left[ -\lambda a \right] &\text{(Assumption)} \\
&= \exp \left[ \left( \lambda - a \right)^{2} /2 \right] \exp \left[ -a^{2}/2 \right]\\
&= \exp \left[ - a^{2}/2 \right] &\text{(Fix $\lambda :=a$)}
\end{aligned}\]
[/latex]

Front: [latex]
Let $X$ be $\No\left( 0, 1 \right)$-distributed. Show that \[
P\left( X \ge a \right) \le \exp \left[ -a^{2}/2 \right].
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item First show $\Exnb \exp \left[ \lambda X \right] = \exp \left[ \lambda ^{2}/2 \right].$ Just do the integral
\[\begin{aligned}
\Exnb \exp \left[ \lambda X \right] &= \int_{}^{} \exp \left[ \lambda x \right] \frac{1}{\sqrt{2\pi} } \exp \left[ -\frac{1}{2}x^{2} \right] dx \\
&=\exp \left[ \lambda ^{2}/2 \right]\int_{}^{} \frac{1}{\sqrt{2 \pi} } \exp \left[ - \frac{1}{2}\left( x - \lambda \right)^{2} \right] dx \\
&= \exp \left[ \lambda ^{2}/2 \right].
\end{aligned}\]
\item Then apply Chebyshev and fix $\lambda := a$.
\[\begin{aligned}
P(X \ge a) &\le \frac{\Exnb \exp \left[ \lambda X \right]}{\exp [\lambda a]} &\text{(Chebyshev)} \\
&=\exp \left[ \lambda ^{2}/2 \right] \exp \left[ -\lambda a \right] &\text{(1.)} \\
&= \exp \left[ \left( \lambda - a \right)^{2} /2 \right] \exp \left[ -a^{2}/2 \right]\\
&= \exp \left[ - a^{2}/2 \right] &\text{(Fix $\lambda :=a$)}
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Let $X_1 , \dots , X_n$ be independent, $X_i \sim \No\left( 0, \sigma _i^{2} \right)$ and $b^{2} = \sum_{i = 1}^{n} \sigma _i^{2}$. Show that
\[\begin{aligned}
\P \left( \sum_{i = 1}^{n} X_i \ge b\sqrt{2t} \right) \le e^{-t} &\text{ for all $t > 0$} .
\end{aligned}\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item We know that for $\No\left( 0, 1 \right)$-distributed $X$ it's true that
\[
P\left( X \ge a \right) \le \exp \left[ -a^{2}/2 \right].
\]
\item From properties of normally distributed random variables, we know \[\sum_{i = 1}^{n} X_i \sim \No\left( 0, \sum_{i = 1}^{n} \sigma ^{2}_i \right) \text{ and hence } \frac{\sum_{i = 1}^{n} X_i}{b} \sim \No\left( 0, 1 \right).\]
\item So, we can now show
\[\begin{aligned}
P\left( \sum_{i = 1}^{n} X_i \ge a \right) &= P\left( \frac{\sum_{i = 1}^{n} X_i}{b} \ge \frac{a}{b} \right) \\
&\le \exp \left[ \frac{-a^{2}}{2b^{2}} \right]. &\text{(Step 1.)}
\end{aligned}\]
\item Now we reformulate the above by setting $a := b\sqrt{2t} $ to get
\[
\P \left( \sum_{i = 1}^{n} X_i \ge b\sqrt{2t} \right) \le e^{-t} \text{ for all $t > 0$} .
\]
\end{enumerate}
[/latex]

Front: [latex]
Suppose $Z_1, \dots, Z_N$ are possibly dependent real-valued random variables, and let $Z_j$ be $\No\left( 0, \sigma ^{2}_j \right)$-distributed. Define $\sigma ^{2}:=\max _{1\le j\le N}\sigma ^{2}_j$. Provide an upper bound on
\[
\P\left( \max _{1 \le j \le N} \left| Z_j \right| \ge \sigma \sqrt{2\left( t + \log N \right)} \right)
\] which does not depend on $N$.
[/latex]
Back: [latex]
The key will be to use the inequality $\left( * \right)$ true for all $a > 0$
\[
P\left( \left| Z_j \right| \ge \sigma _ja \right) \le 2e^{-a^{2}/2}.
\]
For all $t > 0$,
\[\begin{aligned}
\P\left( \max _{1 \le j \le N} \left| Z_j \right| \ge \sigma \sqrt{2\left( t + \log N \right)} \right) &\le \sum_{j = 1}^{N} \P\left( \left| Z_j \right| \ge \sigma \sqrt{2\left( t + \log N \right)} \right) &\text{(Union bound)}\\
&\le \sum_{j = 1}^{N} \max _{1\le i\le n} \P\left( \left| Z_i \right| \ge \sigma \sqrt{2\left( t + \log N \right)} \right) \\
&= N \P\left( \left| Z_k \right| \ge \sigma \sqrt{2\left( t + \log N \right)} \right) &\text{(Let $k$ be argmax)} \\
&\le 2N\exp \left[ -\left( t + \log N \right) \right] &\left( * \right) \\
&= 2\exp \left[ -t \right].
\end{aligned}\]
[/latex]

Front: [latex]
Suppose $X_1, \dots, X_n$ are independent real-valued random variables. What does it mean for them to fulfil Hoeffding's condition?
[/latex]
Back: [latex]
They fulfil Hoeffding's condition if
\begin{enumerate}[(i)]
\item for all $i$, $\Exnb X_i = 0$, and
\item they are bounded, i.e. for certain constants $c_i > 0$, $\left| X_i \right|\le c_i.$
\end{enumerate}
[/latex]

Front: [latex]
Let $X_i, \dots, X_n$ be independent real-valued random variables. Suppose that they fulfil Hoeffding's condition, i.e. for all $i$, $\Exnb X_i = 0$, and for certain constants $c_i > 0$, $\left| X_i \right|\le c_i.$ Show that then for all $\lambda >0$,
\[\begin{aligned}
\Exnb \exp \left[ \lambda X_i \right] &\le \frac{1}{2}\exp \left[ -\lambda c_i \right] + \frac{1}{2} \exp \left[ \lambda c_i \right].
\end{aligned}\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item $x \mapsto \exp \left[ \lambda x \right]$ is convex, i.e. for all $0 < \alpha < 1$,
\[
\exp \left[ \alpha \lambda x + \left( 1 - \alpha \right)\lambda y \right] \le \alpha \exp \left[ \lambda x \right] + \left( 1 - \alpha \right)\exp \left[ \lambda y \right].
\]
\item Let $\alpha _i = \frac{c_i - X_i}{2c_i}$. This is a positive quantity, and less than $1$. We can use $\alpha _i$ to dissect $X_i$ by
\begin{enumerate}[(1)]
\item
\[
1 - \alpha _i = \frac{2c_i - \left( c_i - X_i \right)}{2c_i} = \frac{c_i + X_i}{2c_i}.
\]
\item
\[\begin{aligned}
\alpha _i \left( -c_i \right) + \left( 1 - \alpha _i \right)c_i = \frac{- \left( c_i - X_i \right)}{2} + \frac{\left( c_i + X_i \right)}{2} = X_i.
\end{aligned}\]
\end{enumerate}
\item Hence
\[\begin{aligned}
\exp \left[ \lambda X_i \right] &= \exp \left[ -\alpha _i \lambda c_i + \left( 1 - \alpha _i \right) \lambda c_i \right] &\text{(2)} \\
&\le \alpha _i \exp \left[ -\lambda c_i \right] + \left( 1 - \alpha _i \right) \exp \left[ \lambda c_i \right] &\text{(Convexity)}
\end{aligned}\]
\item Now see that
\[
\Exnb \alpha _i = \frac{1}{2} - \Exnb X_i = \frac{1}{2}.
\]
\item So finally, combining the two steps above,
\[
\Exnb \exp \left[ \lambda X_i \right] \le \frac{1}{2}\exp \left[ -\lambda c_i \right] + \frac{1}{2} \exp \left[ \lambda c_i \right].
\]
\end{enumerate}
[/latex]

Front: [latex]
Let $X_i, \dots, X_n$ be independent real-valued random variables. Suppose that they fulfil Hoeffding's condition, i.e. for all $i$, $\Exnb X_i = 0$, and for certain constants $c_i > 0$, $\left| X_i \right|\le c_i.$ Knowing that for all $\lambda >0$,
\[\begin{aligned}
\Exnb \exp \left[ \lambda X_i \right] &\le \frac{1}{2}\exp \left[ -\lambda c_i \right] + \frac{1}{2} \exp \left[ \lambda c_i \right],
\end{aligned}\] show that
\[
\Exnb \exp \left[ \lambda X_i \right] \le \exp \left[ c_i^{2}\lambda ^{2}/2 \right].
\]
[/latex]
Back: [latex]
The key here is to play with the series expansion of $\exp$.
\begin{enumerate}[1.]
\item Write out \[\begin{aligned}
\exp \left[ -x \right] + \exp \left[ x \right] &= \sum_{m = 0}^{\infty} \frac{\left( -x \right)^{m}}{m!} + \sum_{m = 0}^{\infty} \frac{x^{m}}{m!} \\
&= 2\sum_{m = 0}^{\infty} \frac{x^{2m}}{\left( 2m \right)!} &\text{(Odd $m$ cancel out)}
\end{aligned}\]
\item See that
\[
(2m)! = 2m \left( 2m - 1 \right) \cdots \left( m + 1 \right)m! \ge 2^{m}m!
\]
\item Therefore, we have
\[
\exp \left[ -x \right] + \exp \left[ x \right] = 2\sum_{m = 0}^{\infty} \frac{x^{2m}}{(2m)!} \le 2\sum_{m = 0}^{\infty} \frac{1}{m!} \left( \frac{x^{2}}{2} \right)^{m} = 2 \exp \left[ x^{2}/2 \right].
\]
\item Finally, we use the given inequality to reach
\[\begin{aligned}
\Exnb \exp \left[ \lambda X_i \right] &\le \frac{1}{2}\exp \left[ -\lambda c_i \right] + \frac{1}{2} \exp \left[ \lambda c_i \right]\\
&\le \exp \left[ c_i^{2}\lambda ^{2}/2 \right].
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Let $X_i, \dots, X_n$ be independent real-valued random variables. Suppose that they fulfil Hoeffding's condition, i.e. for all $i$, $\Exnb X_i = 0$, and for certain constants $c_i > 0$, $\left| X_i \right|\le c_i.$ Sketch a proof that then for all $\lambda >0$,
\[\begin{aligned}
\Exnb \exp \left[ \lambda \sum_{i = 1}^{n} X_i \right] \le \exp \left[ \lambda ^{2}b^{2}/2 \right]
\end{aligned}\] where $b^{2} = \sum_{i = 1}^{n} c_i^{2}$.
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item First, we use convexity of $x \mapsto \exp \left[ \lambda x \right]$ to show that
\[
\Exnb \exp \left[ \lambda X_i \right] \le \frac{1}{2}\exp \left[ -\lambda c_i \right] + \frac{1}{2} \exp \left[ \lambda c_i \right].
\]
\item Second, we combine the above with series expansion of $\exp $ to show
\[
\Exnb \exp \left[ \lambda X_i \right] \le \exp \left[ c_i^{2}\lambda ^{2}/2 \right].
\]
\item Lastly, we use independence
\[\begin{aligned}
\Exnb \exp \left[ \lambda \sum_{ i =1}^{n} X_i \right] &= \Exnb \prod_{i = 1}^{n} \exp \left[ \lambda X_i \right]\\
&\le \prod_{i = 1}^{n} \exp \left[ \lambda ^{2}c_i^{2}/2 \right] \\
&= \exp \left[ \lambda ^{2}b^{2}/2 \right].
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Let $X_i, \dots, X_n$ be independent real-valued random variables. Suppose that they fulfil Hoeffding's condition, i.e. for all $i$, $\Exnb X_i = 0$, and for certain constants $c_i > 0$, $\left| X_i \right|\le c_i.$ Define $b^{2} = \sum_{i = 1}^{n} c_i^{2}$. Prove Hoeffding's inequality, so show that for all $a > 0$,
\[\begin{aligned}
\P\left( \sum_{i = 1}^{n} X_i \ge a \right) &\le \exp \left[ - \frac{a^{2}}{2b^{2}} \right].
\end{aligned}\]
Recall the fact that for all $\lambda > 0$,
\[
\Exnb \exp \left[ \lambda \sum_{i = 1}^{n} X_i \right] \le \exp \left[ \lambda ^{2}b^{2}/2 \right].
\]
[/latex]
Back: [latex]
\[\begin{aligned}
\P \left( \sum_{i = 1}^{n} X_i \ge a \right) &\le \frac{\Exnb \exp \left[ \lambda \sum_{i = 1}^{n} X_i \right]}{\exp \left[ \lambda a \right]} &\text{(Chebyshev with $x \mapsto \exp \left[ \lambda x \right]$)} \\
&\le \frac{\exp \left[ \lambda ^{2}b^{2}/2 \right]}{\exp \left[ \lambda a \right]} &\text{(Given inequality)} \\
&= \exp \left[ \lambda ^{2}b^{2}/2 - \lambda a \right]\\
&= \exp \left[ \frac{1}{2} \left[ \lambda b - \frac{a}{b} \right]^{2} \right] \exp \left[ -\frac{a^{2}}{2b^{2}} \right]\\
&= \exp \left[ - \frac{a^{2}}{2b^{2}} \right] &\text{(Fix $\lambda = \frac{a}{b^{2}}$)}
\end{aligned}\]
[/latex]

Front: [latex]
Suppose $X_1, \dots, X_n$ are independent real-valued random variables. When can you use Hoeffding's inequality on them?
[/latex]
Back: [latex]
When they fulfil Hoeffding's condition, i.e. if
\begin{enumerate}[(i)]
\item for all $i$, $\Exnb X_i = 0$, and
\item they are bounded, i.e. for certain constants $c_i > 0$, $\left| X_i \right|\le c_i.$
\end{enumerate}
[/latex]

Front: [latex]
State Hoeffding's inequality.
[/latex]
Back: [latex]
Let $X_i, \dots, X_n$ be independent real-valued random variables. Suppose that they fulfil Hoeffding's condition, i.e. for all $i$, $\Exnb X_i = 0$, and for certain constants $c_i > 0$, $\left| X_i \right|\le c_i.$ Define $b^{2} = \sum_{i = 1}^{n} c_i^{2}$. Then for all $a > 0$,
\[\begin{aligned}
\P\left( \sum_{i = 1}^{n} X_i \ge a \right) &\le \exp \left[ - \frac{a^{2}}{2b^{2}} \right].
\end{aligned}\]
[/latex]

Front: [latex]
Consider a set $\G_N$ of functions $g_j : \X \to \R$, $j = 1, \dots, N$ with, for some constant $K$,
\[
\sup_x \left| g_j(x) \right| \le K.
\]
Show that then for all $t > 0$,
\[
\P\left( \left| \left( \hat{P}_n - P \right) g_j \right| \ge 2K\sqrt{\frac{2t}{n}} \right) \le 2 \exp \left[ -t \right].
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item Note that $\left( \hat{P}_n -P \right)$ fulfils the Hoeffding condition, because
\begin{enumerate}[(1)]
\item
\[
\left( \hat{P}_n - P \right)g_j = \frac{1}{n}\sum_{i = 1}^{n} \left[ g_j(X_i) - \Exnb g_j(X_i) \right]
\] where $\Exnb \left[ g_j(X_i) - \Exnb g_j(X_i) \right] = 0$, and
\item $g_j(X_i)$ are bounded by assumption, meaning that the Hoeffding's inequality $b^{2} = \sum_{i = 1}^{n} \left( 2K/n \right)^{2} = 4K^{2}n$.
\end{enumerate}
\item We can now apply Hoeffding's inequality
\[\begin{aligned}
\P\left( \left( \hat{P}_n - P \right) g_j \ge 2K\sqrt{\frac{2t}{n}} \right) &= \P\left(\sum_{i = 1}^{n} \left[ g_j(X_i) - \Exnb g_j(X_i) \right] \ge 2K\sqrt{2tn} \right) \\
&\le \exp \left[ \frac{-\left( 2K\sqrt{2tn} \right)^{2}}{4nK^{2}} \right] = \exp \left[ -t \right].
\end{aligned}\]
\item Now deal with the absolute value. Since for any random variable $Z$ we have \[\P\left( \left| Z \right| > a \right) \le \P\left( Z > a \right) + \P\left( -Z < -a \right)\] by union bound. So
\[\begin{aligned}
\P\left( \left| \left( \hat{P}_n - P \right) g_j \right| \ge 2K\sqrt{\frac{2t}{n}} \right) &\le 2 \P\left(\left( \hat{P}_n - P \right) g_j \ge 2K\sqrt{\frac{2t}{n}} \right)\\
&\le 2\exp \left[ -t \right].
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Consider a set $\G_N$ of functions $g_j : \X \to \R$, $j = 1, \dots, N$ with, for some constant $K$,
\[
\sup_x \left| g_j(x) \right| \le K.
\]
Sketch key steps of a proof that for all $t > 0$,
\[\begin{aligned}
\P\left( \left\Vert \hat{P}_n - P \right\Vert _{\G_N} \ge 2K \sqrt{\frac{2\left( t + \log \left( 2N \right) \right)}{n}} \right)&\le \exp \left[ -t \right].
\end{aligned}\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item First we need to show an inequality for individual functions $g_j$, so for all $t > 0$,
\[
\P\left( \left| \left( \hat{P}_n - P \right) g_j \right| \ge 2K\sqrt{\frac{2t}{n}} \right) \le 2\exp \left[ -t \right].
\] We can do this by applying Hoeffding's inequality and afterwards accounting for the absolute value.
\item Then we use the above and union bound it. Recall that \[\left\Vert \hat{P}_n - P \right\Vert _{\G_N} :=\max _{g_j \in \G} \left| \left( \hat{P}_n - P \right)g_j \right| \] so
\[\begin{aligned}
\P\bigg( \left\Vert \hat{P}_n - P \right\Vert _{\G_N} &\ge 2K \sqrt{\frac{2\left( t + \log \left( 2N \right) \right)}{n}} \bigg) \\
&\le \sum_{j = 1}^{N} \P\left( \left| \left( \hat{P}_n - P \right)g_j \right|\ge 2K \sqrt{\frac{2\left( t + \log \left( 2N \right) \right)}{n}} \right) &\text{(Union bound)} \\
&\le 2N \exp \left[ -t - \log \left( 2N \right) \right] &\text{(1.)} \\
&= \exp \left[ -t \right].
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Consider a set $\G_N$ of functions $g_j : \X \to \R$, $j = 1, \dots, N$ with, for some constant $K$,
\[
\sup_x \left| g_j(x) \right| \le K.
\]
Show that then for all $j$
\[
\Exnb \exp\left[ \lambda \left| \left( \hat{P}_n - P \right) g_j\right| \right] \le 2 \exp \left[ \frac{\lambda ^{2}\left( 2K \right)^{2}}{2n} \right].
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item
Note that $\left( \hat{P}_n - P \right) g_j$ fulfils the Hoeffding condition, since it's a sum of $n$ centered i.i.d. variables bounded by $2K/n$. So, we can use a lemma which states for any $\lambda > 0$,
\[
\Exnb \exp \left[ \lambda \left( \hat{P}_n - P \right) g_j \right] \le \exp \left[ \frac{\lambda ^{2}\left( 2K \right)^{2}}{2n} \right].
\]
\item So, we deal with the absolute value by $e^{\left| x \right|}\le e^{x}+e^{-x}$ and then apply above lemma
\[\begin{aligned}
\Exnb \exp\left[ \lambda \left| \left( \hat{P}_n - P \right) g_j\right| \right] &\le \Exnb \exp\left[ \lambda \left( \hat{P}_n - P \right) g_j \right] + \Exnb \exp\left[ - \lambda \left( \hat{P}_n - P \right) g_j \right] \\
&\le 2 \exp \left[ \frac{\lambda ^{2}\left( 2K \right)^{2}}{2n} \right].
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Consider a set $\G_N$ of functions $g_j : \X \to \R$, $j = 1, \dots, N$ with, for some constant $K$,
\[
\sup_x \left| g_j(x) \right| \le K.
\] Knowing that for all $j$
\[
\Exnb \exp\left[ \lambda \left| \left( \hat{P}_n - P \right) g_j\right| \right] \le 2 \exp \left[ \frac{\lambda ^{2}\left( 2K \right)^{2}}{2n} \right],
\] show that
\[
\Exnb \left\Vert \hat{P}_n - P \right\Vert \le 2K \sqrt{\frac{2\log \left( 2N \right)}{n}} .
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item We first introduce a $\lambda $ which we will later use as a little optimization hack.
\[\begin{aligned}
\Exnb \left\Vert \hat{P}_n -P \right\Vert &= \frac{1}{\lambda }\Exnb \log \exp \left[ \lambda \left\Vert \hat{P}_n -P \right\Vert \right] \\
&\le \frac{1}{\lambda }\log \Exnb \exp \left[ \lambda \left\Vert \hat{P}_n -P \right\Vert \right]&\text{(Jensen)} \\
&\le \frac{1}{\lambda } \log\sum_{j = 1}^{N} \Exnb \exp \left[ \lambda \left| \left( \hat{P}_n -P \right)g_j \right| \right] &\text{(All positive)} \\
&\le \frac{1}{\lambda }\log \left( 2N \exp \left[ \frac{\lambda ^{2}\left( 2K \right)^{2}}{2n} \right] \right) &\text{(Given inequality)} \\
&=\frac{\log \left( 2N \right)}{\lambda } + \frac{\lambda \left( 2K \right)^{2}}{2n}.
\end{aligned}\]
\item
Then we optimize over $\lambda $ to find the lowest possible upper bound. Simply compute the derivative and set it to 0
\[
-\frac{\log \left( 2N \right)}{\lambda ^{2}} + \frac{\left( 2K \right)^{2}}{2n} \triangleq 0 \implies \lambda = \frac{\sqrt{2n\log \left( 2N \right)} }{2K}.
\]
\item Hence, we get the desired
\[
\Exnb \left\Vert \hat{P}_n - P \right\Vert \le 2K \sqrt{\frac{2\log \left( 2N \right)}{n}} .
\]
\end{enumerate}
[/latex]

Front: [latex]
Consider a set $\G_N$ of functions $g_j : \X \to \R$, $j = 1, \dots, N$ with, for some constant $K$,
\[
\sup_x \left| g_j(x) \right| \le K.
\]
Sketch a proof to show that then
\[
\Exnb \left\Vert \hat{P}_n - P \right\Vert \le K \sqrt{\frac{2\log \left( 2N \right)}{n}} .
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item We first need to show the helper inequality
\[
\Exnb \exp\left[ \lambda \left| \left( \hat{P}_n - P \right) g_j\right| \right] \le 2 \exp \left[ \frac{\lambda ^{2}\left( 2K \right)^{2}}{2n} \right],
\] by using the Hoeffding condition and $e^{\left| x \right|}\le e^{x}+e^{-x}$.
\item Then we introduce $\lambda $, use Jensen, as well as the above inequality to reach a formula to optimize.
\[\begin{aligned}
\Exnb \left\Vert \hat{P}_n -P \right\Vert &= \frac{1}{\lambda }\Exnb \log \exp \left[ \lambda \left\Vert \hat{P}_n -P \right\Vert \right] \\
&\le \frac{1}{\lambda }\log \Exnb \exp \left[ \lambda \left\Vert \hat{P}_n -P \right\Vert \right]&\text{(Jensen)} \\
&\le \frac{1}{\lambda } \log\sum_{j = 1}^{N} \Exnb \exp \left[ \lambda \left| \left( \hat{P}_n -P \right)g_j \right| \right] &\text{(All positive)} \\
&\le \frac{1}{\lambda }\log \left( 2N \exp \left[ \frac{\lambda ^{2}\left( 2K \right)^{2}}{2n} \right] \right) &\text{(Inequality from 1.)} \\
&=\frac{\log \left( 2N \right)}{\lambda } + \frac{\lambda \left( 2K \right)^{2}}{2n}
\end{aligned}\]
\item
Then we optimize over $\lambda $ to find the lowest possible upper bound. Simply compute the derivative and set it to 0
\[
-\frac{\log \left( 2N \right)}{\lambda ^{2}} + \frac{\left( 2K \right)^{2}}{2n} \triangleq 0 \implies \lambda = \frac{\sqrt{2n\log \left( 2N \right)} }{2K}.
\]
\item Hence, we get the desired
\[
\Exnb \left\Vert \hat{P}_n - P \right\Vert \le 2K \sqrt{\frac{2\log \left( 2N \right)}{n}} .
\]
\end{enumerate}
[/latex]

Front: [latex]
Show that for all $t > 0$ and for all $x \in \R$,
\[
\P \left( \left| \hat{F}_n (x) - F(x) \right| > \sqrt{\frac{2t}{n}} \right) \le 2\exp \left[ -t \right].
\]
[/latex]
Back: [latex]
Take $g_x = 1_{\left( -\infty, x \right]}.$ Now we can write the familiar notation
\[
\left| \hat{F}_n(x) - F(x) \right| = \left| \left( \hat{P}_n - P \right)g_x \right|
\] which shows us that we can use Hoeffding's inequality here with $a := \sqrt{\frac{2t}{n}} $ and $b^{2} := 1/n$, and multiplying by 2 to correct the absolute value.
\[\begin{aligned}
\mathbb{P} \left( \left| \left( \hat{P}_n - P \right)g_x \right| \ge \sqrt{\frac{2t}{n}} \right) \le 2 \exp \left[ -\frac{a^{2}}{2b^{2}} \right] = 2 \exp \left[ -t \right]
\end{aligned}\]
[/latex]

Front: [latex]
Consider a set $\G_N$ of functions $g_j : \X \to \R$, $j = 1, \dots, N$ with, for some constant $K$,
\[
\sup_x \left| g_j(x) \right| \le K.
\] Hoeffding's inequality then tells us that for all $t > 0$,
\[
\P\left( \left| \left( \hat{P}_n - P \right) g_j \right| \ge 2K\sqrt{\frac{2t}{n}} \right) \le 2 \exp \left[ -t \right].
\] How does the right side of the inequality inside $\mathbb{P} $ need to change if we replace the left hand side by a maximum over $j$ as
\[
\P\left( \max _{1 \le j \le N}\left| \left( \hat{P}_n - P \right) g_j \right| \ge \left[ ??? \right] \right) \le 2 \exp \left[ -t \right].
\] while keeping the probability bounded by $2\exp \left[ -t \right]$?
[/latex]
Back: [latex]
We need to perform a union bound over all $N$, which adds a $\log \left( N \right)$ term
\[
\P\left( \max _{1 \le j \le N}\left| \left( \hat{P}_n - P \right) g_j \right| \ge 2K\sqrt{\frac{2\left( t + \log \left( N \right) \right)}{n}} \right) \le 2 \exp \left[ -t \right].
\]
[/latex]

Front: [latex]
Let $F$ be a distribution function. Knowing that for all $t > 0$ and for all $x \in \R$
\[
\P \left( \left| \hat{F}_n (x) - F(x) \right| > \sqrt{\frac{2t}{n}} \right) \le 2\exp \left[ -t \right],
\] show that for continuous $F$
\[
\sup _x \left| \hat{F}_n(x) - F(x) \right|\xrightarrow[n \to \infty]{\P} 0.
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item Let $0 < \delta < 1$ be arbitrary. We want to show
\[
\P \left( \left| \hat{F}_n (x) - F(x) \right| > \delta \right) \xrightarrow[n \to \infty]{} 0.
\]
\item We will perform bracketing, so we will bound the supremum by finitely many terms with probabilities that go to 0 as $n \to \infty$. Take $a_0 < a_1 < \cdots < a_N$ such that $F\left( a_j \right) - F\left( a_{j - 1} \right) = \delta $ for $j = 1, \dots, N - 1$. Note that $N \le 1 + 1 / \delta $.
\begin{center}\includegraphics[width=0.8\textwidth]{ept_bracketing_delta}\end{center}
\item Note now that we can bracket any indicator function $1_{\left( -\infty, x \right]}$ by
\[
1_{\left( -\infty, a_{j - 1} \right)} \le 1_{\left( -\infty, x \right]} \le 1_{\left( -\infty, a_j \right]}.
\]
\begin{center}\includegraphics[width=0.8\textwidth]{ept_bracketing_indicators}\end{center}
\item Now we can bound $\hat{F}_n(x) - F(x) $ from above and below.
\[\begin{aligned}
\hat{F}_n(x) - F(x) &\le \hat{F}_n(a_j) - F(x) \\
&\le \hat{F}_n \left( a_j \right) - F\left( a_{j - 1} \right)\\
&= \hat{F}_n \left( a_j \right) - F\left( a_j \right) + \delta, \\
\hat{F}_n(x) - F(x) &\ge \hat{F}_n(a_{j - 1}) - F(x) \\
&\ge \hat{F}_n(a_{j - 1}) - F(a)\\
&= \hat{F}_n(a_{j - 1}) - F(a_{j - 1}) - \delta .
\end{aligned}\]
\item Observe that
\[
\sup _{x} \left| \hat{F}_n(x) - F(x) \right| \le \max _{1 \le j \le N} \left| \hat{F}_n(a_j) - F(a_j) \right| + \delta ,
\] so we can use Hoeffding with an union bound to produce for all $t > 0$
\[
\P \left( \max _{1 \le j \le N} \left| \hat{F}_n(a_j) - F(a_j) \right| > \sqrt{\frac{2\left( t + \log \left( 2N \right) \right)}{n}} \right) \le \exp \left[ -t \right].
\]
\item We now want to show that the right hand side term inside $\mathbb{P} $ is less than $\delta $. Take $t = \frac{n\delta ^{2}}{4}$, and $n \ge \frac{\log \left( 1 + \frac{1}{\delta } \right)}{\delta ^{2}}$. Then
\[
\sqrt{\frac{2\left( t + \log \left( 2N \right) \right)}{n}} \le \sqrt{2 \left( \frac{\delta ^{2}}{4} + \frac{\delta ^{2}}{4} \right)} = \delta.
\]
\item So, for $n \ge 4 \log \left( 2 \left( 1 + \frac{1}{\delta } \right) \right)/\delta ^{2}$ we get
\[\begin{aligned}
\P \left( \sup _{x} \left| \hat{F}_n(x) - F(x) \right| > 2\delta \right) &\le \exp \left[ - \frac{n\delta ^{2}}{4} \right]\\
&\xrightarrow[n \to \infty]{} 0.
\end{aligned}\]
\end{enumerate}
This is the classical Glivenko-Cantelli theorem.
[/latex]

Front: [latex]
Let $\left( S, d \right)$ be a metric space. What is a $\delta $-covering number $N\left( \delta , S, d \right)$ of $S$? What is the entropy of $S$?
[/latex]
Back: [latex]
The $\delta $-covering number $N\left( \delta , S, d \right)$ is the minimum number of balls with radius $\delta $ necessary to cover $S$. Entropy is then defined as
\[
H \left( \cdot, S, d \right) = \log N\left( \cdot, S, d \right).
\]
\begin{center}\includegraphics[width=0.8\textwidth]{ept_covering_number}\end{center}
[/latex]

Front: [latex]
What is an $L_q\left( P \right)$ space? What is its norm $\left\Vert g \right\Vert _q$? What are also $L_\infty$ and $\left\Vert g \right\Vert _\infty $?
[/latex]
Back: [latex]
$L_q\left( P \right)$ is a space of functions $g$ which have a moment of $q$, i.e.
\[
L_q\left( P \right) := \left\{ g : \mathcal{X} \to \mathbb{R} : P \left| g \right|^{q}< \infty \right\}.
\] The norm is then
\[
\left\Vert g \right\Vert _q := \left( P\left| g \right|^{q} \right)^{1/q}.
\]
We also define
\[
L_\infty = \left\{ g : \mathcal{X} \to \mathbb{R} : \sup _{x} \left| g(x) \right| < \infty \right\}
\] and
\[
\left\Vert g \right\Vert _\infty = \sup _{x \in \mathcal{X} } \left| g\left( x \right) \right|.
\]
[/latex]

Front: [latex]
Let $1 \le q < \infty$ and $\G \subset L_q \left( P \right)$ be a set of functions in $L_q$. What is then a $\delta $-bracketing set? What are the $\delta $-covering number with bracketing and entropy with bracketing?
[/latex]
Back: [latex]
For $\delta > 0$, let $\left\{ \left[ g_j^{L}, g_j^{U} \right] \right\}_{j = 1}^{N} \subset L_q\left( P \right)$ be such that
\begin{enumerate}[(i)]
\item $g_j^{U} \ge g_j^{L}$, $\left\Vert g_j^{U} - g_j^{L} \right\Vert _{q} \le \delta $ for all $j$,
\item for all $g \in \G$ there exists a $j$ such that $g_j^{L}\le g \le g_j^{U}$.
\end{enumerate}
Then $\left\{ \left[ g_j^{L}, g_j^{U} \right] \right\}_j^{N}$ is called a $\delta $-bracketing set.
The $\delta $-covering number with bracketing is
\[
N_{q, B}\left( \delta , \mathcal{G} , P \right) := \min \left\{ N : \exists \delta \text{-bracketing set } \left\{ \left[ g_j^{L}, g_j^{U} \right] \right\}_{j = 1}^{N} \right\}.
\]
The entropy with bracketing is analogously $H_{q, B}\left( \delta , \mathcal{G} , P \right) = \log N_{q, B} \left( \delta , S, P \right)$.
\begin{center}\includegraphics[width=0.8\textwidth]{ept_delta_bracketing_set}\end{center}
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $. How is the sup norm entropy $H_\infty \left( \delta , \mathcal{G} \right)$ defined? How is it different from bracketing entropy $H_{q, B}\left( \delta , \mathcal{G} , P \right)$?
[/latex]
Back: [latex]
The sup norm entropy is the entropy $H\left( \cdot, \mathcal{G} , \left\Vert \cdot \right\Vert _\infty \right)$ of $\mathcal{G} $ endowed with the metric induced by the sup norm $\left\Vert \cdot \right\Vert _\infty$. So, it is the $\log $ of the smallest number of functions $g_j$ that fulfil $\left\Vert g - g_j \right\Vert _\infty \le \delta $.
In contrast to the bracketing entropy, it does not depend on the probability measure $P$. Furthermore, the bracketing entropy $H_{q, B}\left( \delta , \mathcal{G} , P \right)$ is based on the number of upper and lower bound functions with $\left\Vert g_j^{U} - g_j^{L} \right\Vert \le \delta $, so the "radius" covered by functions $g_j^{U} - g_j^{L}$ is half of the radius covered by $g_j - \delta, g_j + \delta $ from the $H_\infty$ entropy.
[/latex]

Front: [latex]
Sketch a proof to show that if the entropy with bracketing for a class of functions $\mathcal{G} $ is finite
\[H_{1, B}\left( \cdot, \mathcal{G} , P \right) < \infty\] then this class is Glivenko-Cantelli
\[
\left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} \xrightarrow[n \to \infty]{\mathbb{P} } 0.
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item Fix $\delta > 0$ and pick a bracketing set $\left\Vert g_j^{U} - g_j^{L} \right\Vert _1 \le \delta $.
\item Then we can bound the empirical measure minus the true measure
\[\begin{aligned}
\left( \hat{P}_n - P \right)g &\le \hat{P}_n g_{j}^{U} - Pg\\
&=\left( \hat{P}_n - P \right)g_{j}^{U} + \underbrace{P\left( g_j^{U}- g \right)}_{\le \delta }\\
\left( \hat{P}_n - P \right)g &\ge \hat{P}_n g_j^{L} - Pg \\
&= \left( \hat{P}_n -P \right)g_j^{L} - \underbrace{P\left( g - g_j^{L}\right)}_{\le \delta } .
\end{aligned}\]
\item Then the supremum over $\mathcal{G} $ can be bounded by a max over finitely many functions
\[\begin{aligned}
\left\Vert \hat{P}_n -P \right\Vert _\G \le \underbrace{\max _{1 \le j \le n} \left\{ \left| \left( \hat{P}_n -P \right)g_j^{U} \right| \vee \left| \left( \hat{P}_n -P \right)g_j^{L} \right| \right\}}_{\xrightarrow[n \to \infty]{\P} 0} + \delta .
\end{aligned}\] ($a \vee b = \max \left\{ a, b \right\}$) where we know this limit goes to 0 since it's for a finite class of functions.
\end{enumerate}
[/latex]

Front: [latex]
Suppose $\mathcal{G}$ is a class of functions. What is an envelope function of $\mathcal{G} $? What does it mean for $\mathcal{G} $ to be uniformly bounded?
[/latex]
Back: [latex]
The envelope function $G$ is
\[
G := \sup _{g \in \mathcal{G} }\left| g \right|.
\] $\mathcal{G} $ is uniformly bounded if the infinity norm of the envelope is finite
\[
\left\Vert G \right\Vert _\infty < \infty.
\]
[/latex]

Front: [latex]
Suppose $\mathcal{G}$ is a class of functions in $L_q\left( P \right)$. What does finiteness of entropies $H_q\left( \cdot, \mathcal{G} , P \right)$ or $H_\infty \left( \cdot, \mathcal{G} \right)$ imply about the envelope of $\mathcal{G} $?
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item $H_q\left( \cdot, \mathcal{G} , P \right) < \infty \implies G \in L_q\left( P \right)$,
\item $H_\infty \left( \cdot, \mathcal{G} \right) < \infty \implies \left\Vert G \right\Vert _\infty < \infty$.
\end{enumerate}
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G}$. Why is it preferable for us to prove statements about the entropy of $\mathcal{G} $ in infinity norm $L_\infty$ as opposed to $L_q$ norms?
[/latex]
Back: [latex]
$L_q$ norms depend on the probability measure $P$, while $L_\infty$ does not. So, if we can prove statements using $L_\infty$, we prove them independent of the underlying probability measure.
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $. Show that for all $1 \le q < \infty$, we can bound the entropy with bracketing by the entropy with sup norm \[H_{q, B} \left( 2\delta , \G, P \right) \le H_\infty \left( \delta , \G \right)\] for all $\delta $.
[/latex]
Back: [latex]
Assume $H_\infty\left( \delta , \mathcal{G} \right) < \infty$; otherwise, the inequality is trivially true.
Let $\left\{ g_j \right\}_{j = 1}^{N}$ be such that for all $g \in \G$ there exists a $j$ such that $\left\Vert g - g_j \right\Vert _{\infty} \le \delta $. Let $g_j^{U} := g_j + \delta $, $g_j^{L} = g_j - \delta $.
\begin{center}\includegraphics[width=0.8\textwidth]{ept_delta_brackets}\end{center}
Then $g_j^{L} \le g \le g_j^{U}$ and $\left\Vert g_j^{U} - g_{j}^{L} \right\Vert _\infty \le 2\delta $. Also, since the total measure of the integral in $\left\Vert g \right\Vert _q$ is 1, we can tell that all $g$ \[\left\Vert g \right\Vert _q \le \left\Vert g \right\Vert _\infty.\]
This means that there exists a $2\delta $-bracketing set $N_{q, B}\left( 2\delta , \mathcal{G} , P \right)$ in $L_q$, which is smaller than $N_{\infty}\left( \delta , \mathcal{G} \right)$.
[/latex]

Front: [latex]
Let $\mathcal{G} $ be the class of bounded Lipschitz functions
\[
\G := \left\{ g: \left[ 0, 1 \right]\to \left[ 0, 1 \right], \left| g(x) - g(\tilde{x}) \right| \le \left| x - \tilde{x} \right| \text{ for all $x, \tilde{x}$} \right\}.
\] Find an upper bound for $H_\infty\left( \delta , \mathcal{G} \right)$.
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item Consider a discretization of the unit square. Let $a_0 < a_1 < \dots < a_{n}$, $a_j - a_{j - 1} \le \delta $, $N \le 1 + \frac{1}{\delta }$.
\begin{center}\includegraphics[width=0.8\textwidth]{ept_discretization_square}\end{center}
\item
For $g \in \G$, $x \in \left( a_{j - 1}, a_j \right]$ take a discretized $g_j$
\[
g_j(x) = \left\lfloor \frac{g\left( a_j \right)}{\delta } \right\rfloor \delta .
\]
\item We want to find the radius of the cover produced by this construction, i.e. how large the maximum absolute difference between $g(x)$ and $g_j(x)$ can be, as well as an upper bound on the number of possible $g_j(x)$.
\item To find the maximum difference, first note that
\[
\frac{g\left( a_j \right)}{\delta } - 1 < \left\lfloor \frac{g\left( a_j \right)}{\delta } \right\rfloor \le \frac{g\left( a_j \right)}{\delta }.
\] Now we can write
\[\begin{aligned}
g(x) - g_{j}(x) &= g(x) - \left\lfloor \frac{g(a_j)}{\delta } \right\rfloor \delta \\
&< g(x) - \left( \frac{g(a_j)}{\delta } - 1\right)\delta \\
&= g(x) - g(a_j) + \delta \\
&< 2\delta &\text{(Lipschitz)}, \\
g(x) - g_{j}(x) &= g(x) - \left\lfloor \frac{g(a_j)}{\delta } \right\rfloor \delta \\
&\ge g(x) - \left( \frac{g(a_j)}{\delta }\right)\delta \\
&= g(x) - g(a_j) \\
&\ge -\delta &\text{(Lipschitz)}
\end{aligned}\] so $\left| g(x) - g_j(x) \right| < 2\delta $, meaning that this method is appropriate to find a $2\delta $-covering.
\item To find how many possible $g_j(x)$ there are, we want to bound the possible number of values of $\left\lfloor \frac{g(a_j)}{\delta } \right\rfloor $ given $\left\lfloor \frac{g(a_{j - 1})}{\delta } \right\rfloor $. We will do a conservative upper bound, because we only need to show that it exists and not that it is tight.
\[\begin{aligned}
\left| \left\lfloor \frac{g(a_j)}{\delta } \right\rfloor - \left\lfloor \frac{g(a_{j - 1})}{\delta } \right\rfloor \right| &\le 2 + \left| \frac{g\left( a_j \right)}{\delta } - \frac{g\left( a_{j - 1} \right)}{\delta } \right| \stackrel{\text{Lipschitz} }{\le } 3
\end{aligned}\]
So $g_j$ can jump at most three up, three down, or stay constant, giving us 7 possibilities per step. The number of choices for $g_j\left( a_1 \right)$ is at most $1 + \frac{1}{\delta }$, so
\[
\#g_j \le \left( 1 + \frac{1}{\delta } \right) \underbrace{7 \times \cdots \times 7}_{N - 1 \le \frac{1}{\delta }} \le \left( 1 + \frac{1}{\delta } \right)7^{1/\delta }.
\]
\item Now just $\log $ the above to find a bound for the entropy
\[\begin{aligned}
H_\infty \left( 2\delta , \G \right) &\le \frac{1}{\delta } \log 7 + \log \left( 1 + \frac{1}{\delta } \right)\\
&\le \frac{1}{\delta } \log 7 + \frac{1}{\delta }\\
& = \frac{1}{\delta } \left( \log 7 + 1 \right).
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Show that if $H_{q, B}\left( \cdot, \G, P \right) < \infty$, then the envelope $G \in L_q\left( P \right)$.
[/latex]
Back: [latex]
If the entropy with bracketing is finite, then there exists a finite bracketing set. Then for all $g \in \G$,
\[\begin{aligned}
g &\le g_j^{U} \le \left| g_j^{U} \right| \le \sum_{j = 1}^{N} \left| g_j^{U} \right| \\
g &\ge g_j^{L} \ge -\left| g_j^{L} \right| \ge -\sum_{j = 1}^{N} \left| g_j^{L} \right|
\end{aligned}\]
Then
\[\begin{aligned}
G &\le \sum_{j = 1}^{N} \left\{ \left| g_j^{U} \right| + \left| g_j^{L} \right| \right\} \in L_q.
\end{aligned}\]
[/latex]

Front: [latex]
Let $\mathcal{G} := \left\{ g_\theta : \theta \in \Theta \right\}$. Suppose
\begin{enumerate}[(i)]
\item $\left( \Theta ,d \right)$ is a compact metric space,
\item $\theta \mapsto g_\theta (x)$ is continuous for all $x \in \mathcal{X} $, and
\item there exists an envelope $G\left( \cdot \right) := \sup _{\theta \in \Theta } \left| g_\theta \left( \cdot \right) \right| \in L_q\left( P \right)$.
\end{enumerate}
Show that then the entropy with bracketing is finite
\[
H_{q, B}\left( \cdot, \mathcal{G} , P \right) < \infty.
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item Let $\delta > 0$, take a radius $\rho > 0$, and define the pointwise supremum
\[
w_{\theta , \rho} \left( x \right) := \sup _{\tilde{\theta }: d\left( \tilde{\theta }, \theta \right) < \rho} \left| g_\theta(x) -g_{\tilde{\theta }} (x) \right|.
\]
\item Then as we take the limit of $\rho \to 0$, by condition $(ii)$ (pointwise continuity) we get
\[\begin{aligned}
\lim_{\rho \to 0} w_{\theta , \rho }\left( x \right) = 0.
\end{aligned}\]
\item Since the limit is 0 for each point and the envelope is in $L_q\left( P \right)$ (iii), we can use dominated convergence to show that on average the limit is 0, i.e.
\[\begin{aligned}
P w_{\theta , \rho }^{q} &\xrightarrow[\rho \to 0]{} 0. \end{aligned}\]
\item This implies that there exists a radius $\rho _\theta $ depending on $\theta $ such that
\[
Pw_{\theta, \rho _\theta}^{q} \le \delta^{q}.
\]
\item
Now define a cover of $\Theta $ with balls $B_\theta = \left\{ \tilde{\theta }: d\left( \theta , \tilde{\theta } \right) < \rho _\theta \right\}$. $\left\{ B_\theta \right\}_{\theta \in \Theta }$
is an open cover of $\Theta $, which by compactness (i) implies there exists a finite subcover $\left\{ B_{\theta _j} \right\}_{j = 1}^{N}$.
\item Now we can use bracketing to bound $g_\theta $ from above and below using $\theta _j$ from $B_{\theta _j}$. Write $w_j = w_{\theta _j, \rho _{\theta _j}} $ , so then
\[\begin{aligned}
g_\theta &= g_{\theta _j} + g_\theta - g_{\theta _j} \\
&\le \underbrace{g_{\theta _j} + w_j }_{\text{upper function }g_j^U },\\
g_\theta &\ge \underbrace{g_{\theta _j} - w_j }_{\text{lower function }g_j^L }
\end{aligned}\]
\item So, we get
\[\begin{aligned}
P\left( g_j^{U}- g_j^{L} \right)^{q} &=P\left( 2 w_j \right)^{q}\\
&\le 2^{q}\delta ^{q}
\end{aligned}\] and we can say that $g_j^{L}$ and $g_j^{U}$ create a $\delta ^{1/q}$-bracketing set. Since this set is finite, the entropy is also finite, which concludes the proof.
\end{enumerate}
[/latex]

Front: [latex]
Let $\mathcal{G} := \left\{ g_\theta : \theta \in \Theta \right\}$. Give sufficient conditions for $H_{q, B}\left( \cdot, \mathcal{G} , P \right) < \infty$.
[/latex]
Back: [latex]
If
\begin{enumerate}[(1)]
\item $\left( \Theta ,d \right)$ is a compact metric space,
\item $\theta \mapsto g_\theta (x)$ is continuous for all $x \in \mathcal{X} $, and
\item there exists an envelope $G\left( \cdot \right) := \sup _{\theta \in \Theta } \left| g_\theta \left( \cdot \right) \right| \in L_q\left( P \right)$
\end{enumerate}
then $H_{q, B}\left( \cdot, \mathcal{G} , P \right) < \infty$.
[/latex]

Front: [latex]
Let $Y \in \mathbb{R} $ be a random variable with unknown distribution function $F_0$. Given $Y = y$, let $X$ have a known distribution with density $K\left( x | y \right)$. Then $X$ has mixture density
\[
p_{F_0} := \int_{}^{} K\left( \cdot | y \right)dF_0\left( y \right).
\] We observe $n$ i.i.d. copies $X_1, \dots, X_n$ of $X$. Let $\mathcal{F} $ be the collection of all distribution functions. The maximum likelihood estimator of $F_0$ is
\[
\hat{F}_n := \arg \max _{F \in \mathcal{F} }\hat{P}_n \log p_{F},
\] where $\mathcal{F} \subset \mathcal{F} ^{*}$ with $\left( \mathcal{F} ^{*}, d \right)$ compact. Show that the class
\[
\mathcal{G} := \left\{ g_F(x) = \frac{p_F(x)}{p_F(x) + p_{F_0(x)}} : F \in \mathcal{F} ^{*} \right\}
\] is Glivenko-Cantelli.
[/latex]
Back: [latex]
Note that $\mathcal{G} $ satisfies the sufficient conditions such that $H_{1, B}\left( \cdot, \mathcal{G} , P \right)<\infty$, and thus also
\[
\left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} \xrightarrow[n \to \infty]{\mathbb{P} } 0.
\]
To see that it satisfies these conditions, see that
\begin{enumerate}[1.]
\item $\mathcal{F} ^{*}$ is compact,
\item $F \mapsto g_F(x)$ is continuous for all $x \in \mathcal{X} $, and
\item $\sup _{F \in \mathcal{F} ^{*}} \left| g_F \left( x \right) \right| \le 1$, meaning that $G(x) = 1$ is an envelope function with $\left\Vert G \right\Vert _\infty < \infty$, leading to $H_\infty\left( \cdot, \mathcal{G} \right) < \infty$, and ultimately $H_{1, B}\left( \cdot, \mathcal{G} , P \right) < \infty$.
\end{enumerate}
[/latex]

Front: [latex]
Consider two independent samples $\textbf{X} = X_1, \dots, X_n$ and $\textbf{X} ' = X_1', \dots, X_n'$. Let $\hat{P}_n $ and $\hat{P}_n '$ be their corresponding empirical measures. Show that
\[
\Exnb \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} \le \Exnb \left\Vert \hat{P}_n - \hat{P}_n ' \right\Vert _\mathcal{G} .
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item Begin by comparing expectations of $\hat{P}_n g$ and $\hat{P}_n 'g$ conditional on $\textbf{X} $. We have
\[\begin{aligned}
\Exnb \left[ \hat{P}_n g | \textbf{X} \right] &= \hat{P}_n g\\
\Exnb \left[ \hat{P}_n' g | \textbf{X} \right] &= Pg. &\text{(Independence)}
\end{aligned}\]
\item So, we can write
\[
\left( \hat{P}_n - P \right)g = \Exnb \left[ \left( \hat{P}_n - \hat{P}_n ' \right)g | \textbf{X} \right]
\] and ultimately
\[
\left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} = \sup _{g \in \mathcal{G} } \left| \left( \hat{P}_n - P \right)g \right| = \sup _{g \in \mathcal{G} } \left| \Exnb \left[ \left( \hat{P}_n - \hat{P}_n ' \right)g | \textbf{X} \right] \right|.
\]
\item So then we can use properties of expectation to take the $\sup $ into the expectation as
\[\begin{aligned}
\left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} &= \sup _{g \in \mathcal{G} } \left| \Exnb \left[ \left( \hat{P}_n - \hat{P}_n ' \right)g | \textbf{X} \right] \right| \\
&\le \sup _{g \in \mathcal{G} } \Exnb \left[ \left| \left( \hat{P}_n - \hat{P}_n ' \right)g \right| | \textbf{X} \right] &\text{(Jensen)} \\
&\le \Exnb \left[ \left\Vert \hat{P}_n - \hat{P}_n ' \right\Vert _\mathcal{G} | \textbf{X} \right] .
\end{aligned}\]
\item Now combine the above with iterated expectations
\[\begin{aligned}
\Exnb \left[ \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} \right] &\le \Exnb \left[ \Exnb \left( \left\Vert \hat{P}_n - \hat{P}_n ' \right\Vert _{\mathcal{G} } | \textbf{X} \right) \right] \\
&= \Exnb \left[ \left\Vert \hat{P}_n - \hat{P}_n ' \right\Vert _\mathcal{G} \right] .
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Let $Z \in \R$ be symmetric
\[
\P \left( Z \le z \right) = \P\left( Z \ge -z \right) \text{ for all $z$}
\] and let $\sigma \in \left\{ -1, 1 \right\}$, $\P\left( \sigma = 1 \right) = \P\left( \sigma = -1 \right) = \frac{1}{2}$, with $\sigma $ independent of $Z$. Show that then
\[
Z \substack{\D \\ =} \sigma Z
\] meaning that $Z$ and $\sigma Z$ have the same distribution.
[/latex]
Back: [latex]
\[\begin{aligned}
\P \left( \sigma Z \le z \right) &=\P \left( \sigma Z \le z | \sigma = 1 \right)\frac{1}{2} + \P\left( \sigma Z \le z | \sigma = -1 \right) \frac{1}{2}\\
&=\P\left( Z \le z \right) \frac{1}{2} + \P\left( -Z \le z \right) \frac{1}{2}\\
&=\P\left( Z \le z \right)\frac{1}{2} + \P\left( Z \ge -z \right)\frac{1}{2}\\
&=\P\left( Z \le z \right).
\end{aligned}\]
[/latex]

Front: [latex]
What is a Rademacher sequence?
[/latex]
Back: [latex]
A Rademacher sequence is a sequence $\left\{ \sigma _i \right\}_{i = 1}^{n}$ i.i.d. with $\P\left( \sigma _i = 1 \right) = \P\left( \sigma _i = -1 \right) = \frac{1}{2}$.
[/latex]

Front: [latex]
Let $\left\{ \sigma _i \right\}_{i = 1}^{n}$ be a Rademacher sequence independent of the sample $X_1, \dots, X_n$, and $g$ a function. What is the symmetrized empirical measure $\hat{P}_n ^{\sigma }g$?
[/latex]
Back: [latex]
The symmetrized empirical measure $\hat{P}_n ^{\sigma }g$ is defined as
\[
\hat{P}_n ^{\sigma }g := \frac{1}{n} \sum_{i = 1}^{n} \sigma _ig\left( X_i \right).
\]
[/latex]

Front: [latex]
Let $\left\{ \sigma _i \right\}_{i = 1}^{n}$ be a Rademacher sequence independent of $\textbf{X}$. Show that then
\[
\Exnb \left[ \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} \right] \le 2 \Exnb \left[ \left\Vert \hat{P}_n ^{\sigma } \right\Vert _\mathcal{G} \right].
\]
[/latex]
Back: [latex]
First we need to bound $\Exnb \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G}$ by $\Exnb \left\Vert \hat{P}_n -\hat{P}_n ' \right\Vert _\mathcal{G} $ with an independent sample $\textbf{X} '$, and then use symmetrization.
\begin{enumerate}[1.]
\item Consider a sample $\textbf{X} '$ with the same distribution as $\textbf{X} $, but independent of it. Using independence and properties of $\sup $ and Jensen, we can bound $\left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} $ by a conditional expectation
\[\begin{aligned}
\left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} &= \sup _{g \in \mathcal{G} } \left| \Exnb \left[ \left( \hat{P}_n - \hat{P}_n ' \right)g | \textbf{X} \right] \right| \\
&\le \sup _{g \in \mathcal{G} } \Exnb \left[ \left| \left( \hat{P}_n - \hat{P}_n ' \right)g \right| | \textbf{X} \right] &\text{(Jensen)} \\
&\le \Exnb \left[ \left\Vert \hat{P}_n - \hat{P}_n ' \right\Vert _\mathcal{G} | \textbf{X} \right] .
\end{aligned}\]
\item Now we can use iterated expectations on the above to bound $\Exnb \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} $
\[\begin{aligned}
\Exnb \left[ \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} \right] &\le \Exnb \left[ \Exnb \left( \left\Vert \hat{P}_n - \hat{P}_n ' \right\Vert _{\mathcal{G} } | \textbf{X} \right) \right] \\
&= \Exnb \left[ \left\Vert \hat{P}_n - \hat{P}_n ' \right\Vert _\mathcal{G} \right] .
\end{aligned}\]
\item Note that $\left\{ g\left( X_i \right) - g\left( X_i' \right) \right\}_{g \in \G}$ is equal in distribution to $\left\{ \sigma _i \left( g\left( X_i \right) - g\left( X_i' \right) \right) \right\}_{g \in \G}$.
\item So, we get that
\[\begin{aligned}
\Exnb \left\Vert \hat{P}_n - \hat{P}_n ' \right\Vert _{\G} &= \Exnb \left\Vert \hat{P}_n ^{\sigma } - \hat{P}_n'^{\sigma } \right\Vert _\G\\
&\le \Exnb \left\Vert \hat{P}_n ^{\sigma } \right\Vert _\G + \Exnb \left\Vert \hat{P}_n '^{\sigma } \right\Vert _\G\\
&\le 2 \Exnb \left\Vert \hat{P}_n ^{\sigma } \right\Vert _\G.
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Write $\Exnb \left[ \left( \hat{P}_n - P \right)g \right] ^{2}$ in terms of $\var{g(X)} $.
[/latex]
Back: [latex]
\[
\Exnb \left[ \left( \hat{P}_n - P \right)g \right] ^{2} = \frac{1}{n}\var{g(X)}.
\]
This is because
\[\begin{aligned}
\Exnb \left[ \left( \hat{P}_n - P \right)g \right] ^{2} &= \Exnb \left[ \frac{1}{n} \sum_{i = 1}^{n} \left( g\left( X_i \right) - \Exnb g\left( X_i \right) \right)^{2} \right] \\
&= \frac{1}{n^{2}} \sum_{i = 1}^{n} \Exnb \left[ g\left( X_i \right) - \Exnb g\left( X_i \right) \right] ^{2}\\
&= \frac{1}{n}\var{g(X)}.
\end{aligned}\]
[/latex]

Front: [latex]
Let $g : \X \to \R$, $\var{g(X)} \le \frac{n\delta ^{2}}{8}$. Show that then
\[
\P\left( \left| \left( \hat{P}_n - P \right)g \right| > \frac{\delta }{2} \right) \le \frac{1}{2}.
\]
[/latex]
Back: [latex]
This is just a simple application of Chebyshev's inequality with $\phi(x) = x^{2}$, which is increasing for nonnegative $x$.
\[\begin{aligned}
\P\left( \left| \left( \hat{P}_n - P \right)g \right| > \frac{\delta }{2} \right) &\le \frac{\Exnb \left[ \left( \hat{P}_n - P \right)g \right] ^{2}}{\delta ^{2}/4}\\
&=\frac{\var{g(X)} }{n\delta ^{2}/4} \\
&\le \frac{n\delta ^{2}/8}{n\delta ^{2}/4} = \frac{1}{2}
\end{aligned}\]
[/latex]

Front: [latex]
Consider two independent samples $\textbf{X} = X_1, \dots, X_n$ and $\textbf{X} ' = X_1', \dots, X_n'$. Let $\hat{P}_n $ and $\hat{P}_n '$ be their corresponding empirical measures. Suppose for all $g \in \mathcal{G} $,
\[
\mathbb{P} \left( \left| \left( \hat{P}_n - P \right)g \right| > \delta /2 \right) \le \frac{1}{2}.
\]
Show that then
\[
\P \left( \left\Vert \hat{P}_n - P \right\Vert _\G > \delta \right) \le 2 \P\left( \left\Vert \hat{P}_n - \hat{P}_n ' \right\Vert _\G > \frac{\delta }{2} \right).
\]
[/latex]
Back: [latex]
The proof will largely rely on conditional probabilities based on $\textbf{X} $ combined with independence of $\textbf{X} $ and $\textbf{X} '$. We want to be able to compare $\left| \left( \hat{P}_n - P \right)g \right|$ with $\left| \left( \hat{P}_n' - P \right)g' \right|$ even though in the supremum norm $\left\Vert \cdot \right\Vert _\mathcal{G} $ they might \textit{refer to} different functions $g$. So, we need to find an appropriate function $g$ that we can use to compare their norms.
\begin{enumerate}[1.]
\item
Let $\mathbb{P} _\textbf{X} $ denote the conditional probability given $\textbf{X} $. If $\left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} > \delta $, then we know that there exists some function $g^{*} = g^{*}\left( \textbf{X} \right)$ depending on $\textbf{X} $ such that
\[
\left| \left( \hat{P}_n - P \right)g^{*} \right| = \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} > \delta .
\]
\item
Because $\textbf{X} '$ is independent of $\textbf{X} $, from an assumption we know that
\[
\mathbb{P} \left( \left| \left( \hat{P}_n' - P \right)g^{*} \right| > \delta /2 \right) \le \frac{1}{2}.
\]
\item
Hence
\[\begin{aligned}
&\mathbb{P} \left( \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta \text{ and } \left| \left( \hat{P}_n' - P \right)g^{*} \right| \le \frac{\delta }{2} \right) \\
& = \Exnb \mathbb{P} _\textbf{X} \left( \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta \text{ and } \left| \left( \hat{P}_n' - P \right)g^{*} \right| \le \frac{\delta }{2} \right)\\
& = \Exnb \mathbb{P} _\textbf{X} \left( \left| \left( \hat{P}_n '- P \right)g^{*} \right| \le \frac{\delta }{2}\right)1 \left\{ \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta \right\}\\
& \ge \frac{1}{2} \Exnb 1 \left\{ \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta \right\}\\
&= \frac{1}{2}\mathbb{P} \left( \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta \right).
\end{aligned}\]
\item
So we can bound the probability of $\left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} > \delta $ as
\[\begin{aligned}
\mathbb{P} \left( \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} > \delta \right) &\le \mathbb{P} \left( \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta \right) \\
&\le 2 \mathbb{P} \left( \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta \text{ and } \left| \left( \hat{P}_n' - P \right)g^{*} \right| \le \frac{\delta }{2} \right)\\
&\le 2\mathbb{P} \left( \left| \left( \hat{P}_n - \hat{P}_n ' \right)g^{*} \right| > \frac{\delta }{2} \right).
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Consider two independent samples $\textbf{X} = X_1, \dots, X_n$ and $\textbf{X} ' = X_1', \dots, X_n'$. Let $\hat{P}_n $ and $\hat{P}_n '$ be their corresponding empirical measures. Suppose for all $g \in \mathcal{G} $,
\[
\mathbb{P} \left( \left| \left( \hat{P}_n - P \right)g \right| > \delta /2 \right) \le \frac{1}{2}.
\]
Knowing that the above implies
\[
\mathbb{P} \left( \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} > \delta \right) \le 2\mathbb{P} \left( \left\Vert \hat{P}_n - \hat{P}_n ' \right\Vert _\mathcal{G} > \frac{\delta }{2} \right),
\] show that
\[
\mathbb{P} \left( \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} > \delta \right) \le 4 \mathbb{P} \left( \left\Vert \hat{P}_n^{\sigma} \right\Vert _\mathcal{G} > \frac{\delta }{4} \right).
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item Note that for any $\delta > 0$ and any real $X, Y$ it's true that
\[
\left| X - Y \right| > \delta / 2 \implies \left| X \right| > \delta /4 \text{ or } \left| Y \right| > \delta /4.
\]
\item Then, starting from the hint given,
\[\begin{aligned}
\mathbb{P} \left( \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} > \delta \right) &\le 2\mathbb{P} \left( \left\Vert \hat{P}_n - \hat{P}_n ' \right\Vert _\mathcal{G} > \frac{\delta }{2} \right)\\
&= 2\mathbb{P} \left( \left\Vert \hat{P}_n ^{\sigma} - \hat{P}_n '^{\sigma} \right\Vert _{\mathcal{G} } > \frac{\delta }{2} \right)\\
&\le 2\left[ \mathbb{P} \left( \left\Vert \hat{P}_n ^{\sigma} \right\Vert _\mathcal{G} > \frac{\delta }{4} \text{ or } \left\Vert \hat{P}_n '^{\sigma} \right\Vert _\mathcal{G} > \frac{\delta }{4} \right) \right] &\text{(1.)} \\
&\le 2\left[ \mathbb{P} \left( \left\Vert \hat{P}_n ^{\sigma} \right\Vert _\mathcal{G} > \frac{\delta }{4} \right) + \mathbb{P} \left( \left\Vert \hat{P}_n '^{\sigma} \right\Vert _\mathcal{G} > \frac{\delta }{4} \right) \right] &\text{(Union bound)} \\
&= 4\mathbb{P} \left( \left\Vert \hat{P}_n ^{\sigma} \right\Vert_\mathcal{G} > \frac{\delta }{4} \right). &\text{(Same distribution)}
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Let $\left\{ \sigma _i \right\}_{i = 1}^{n}$ be a Rademacher sequence independent of the sample $X_1, \dots, X_n$, and $g$ a function. Derive an upper bound for the conditional probability which vanishes at least exponentially in $t$
\[
\mathbb{P} \left( \left| \hat{P}_n ^{\sigma}g \right| > t \mid \textbf{X} \right).
\]
[/latex]
Back: [latex]
\[
\hat{P}_n ^{\sigma}g \mid \textbf{X} = \frac{1}{n}\sum_{i = 1}^{n} \sigma _i g\left( X_i \right) \text{ with $X_i$ fixed}
\]
$\hat{P}_n ^{\sigma}g \mid \textbf{X}$ satisfies Hoeffding's condition, because for all $i$
\begin{enumerate}[(a)]
\item $\Exnb \sigma _i g\left( X_i \right) = 0$,
\item $\sigma _i g\left( X_i \right)/n \le \sup_{1 \le j \le n} \left| g\left( X_j \right) \right|/n$.
\end{enumerate}
So, we can apply Hoeffding's inequality with $a^{2} = t^{2}$ and, with the notation of \[\left\Vert g \right\Vert _n^{2} = \frac{1}{n}\sum_{i = 1}^{n} g^{2}\left( X_i \right),\] we can write \[b^{2} = \sum_{i = 1}^{n} g^{2}(X_i)/n^{2} = \frac{1}{n}\left\Vert g \right\Vert _n^{2}.\]
Then the bound we get by Hoeffding is
\[
\mathbb{P} \left( \left| \hat{P}_n ^{\sigma}g \right| > t \mid \textbf{X} \right) \le 2\exp \left[ -\frac{nt^{2}}{2\left\Vert g \right\Vert _n^{2}} \right].
\]
[/latex]

Front: [latex]
Let $\left\{ \sigma _i \right\}_{i = 1}^{n}$ be a Rademacher sequence independent of the sample $X_1, \dots, X_n$, and $\mathcal{G} = \left\{ g_1, \dots, g_N \right\}$ a class of bounded functions. What term must you put in for $Z$ for the inequality below to work, assuming Hoeffding's conditions are held?
\[\begin{aligned}
\P\left( \max _{1 \le j \le N} \left| \hat{P}_n ^{\sigma }g_j \right| > Z \right) \le \exp \left[ -t \right].
\end{aligned}\]
[/latex]
Back: [latex]
We must put in
\[
Z = \max _{1 \le j \le N} \left\Vert g_j \right\Vert_n \sqrt{\frac{2\left( t + \log \left( 2N \right) \right)}{n}}
\] where $\left\Vert g_j \right\Vert _n^{2} := \frac{1}{n} \sum_{i = 1}^{n} g^{2}\left( X_i \right)$ such that
\begin{enumerate}[(1)]
\item $\max _{1 \le j \le N} \left\Vert g_j \right\Vert _n$ ensures the denominator in $\exp $ is 1, and
\item $\log \left( 2N \right)$ accounts for the union bound we get by $\max _{1 \le j \le N}$ and factor of 2 caused by absolute value around $\hat{P}_n ^{\sigma}g_j$.
\end{enumerate}This ultimately results in
\[\begin{aligned}
\P\left( \max _{1 \le j \le N} \left| \hat{P}_n ^{\sigma }g_j \right| > \max _{1 \le j \le N} \left\Vert g_j \right\Vert _n \sqrt{\frac{2\left( t + \log \left( 2N \right) \right)}{n}} \right) \le \exp \left[ -t \right].
\end{aligned}\]
[/latex]

Front: [latex]
Let $\left\{ \sigma _i \right\}_{i = 1}^{n}$ be a Rademacher sequence independent of the sample $X_1, \dots, X_n$, and $\mathcal{G} = \left\{ g_1, \dots, g_N \right\}$ a class of bounded functions. Which condition must hold for us to be able to use symmetrization to write
\[\begin{aligned}
\P \left( \left\Vert \hat{P}_n - P \right\Vert _\G > \delta \right) &\le 4 \P \left( \left\Vert \hat{P}_n ^{\sigma } \right\Vert > \frac{\delta }{4}\right)?
\end{aligned}\] Sketch an explanation for why.
[/latex]
Back: [latex]
The above is true if $\var{g\left( X \right)} \le \frac{n\delta ^{2}}{8}$ for all $g \in \G$.
\begin{enumerate}[1.]
\item By Chebyshev, we can show that $\var{g(X)} \le \frac{n\delta ^{2}}{8}$ implies
\[
\P\left( \left| \left( \hat{P}_n - P \right)g \right| > \frac{\delta }{2} \right) \le \frac{1}{2}.
\]
\item We can consider a hypothetical independent sample $\textbf{X} '$ with the same distribution as $\textbf{X} $. If $\left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} > \delta $, then for some random function $g^{*} = g^{*}\left( \textbf{X} \right)$ depending on $\textbf{X} $,
\[
\left| \left( \hat{P}_n - P \right)g^{*} \right| = \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} > \delta .
\]
\item We can then play with step 1. and conditional expectations based on $\textbf{X} $ to show
\[\begin{aligned}
&\mathbb{P} \left( \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta \text{ and } \left| \left( \hat{P}_n' - P \right)g^{*} \right| \le \frac{\delta }{2} \right) &\left( * \right) \\
&=\Exnb \mathbb{P}_\textbf{X} \left( \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta \text{ and } \left| \left( \hat{P}_n' - P \right)g^{*} \right| \le \frac{\delta }{2} \right) &\text{(Conditional $\Exnb $)} \\
&=\mathbb{P} \left( \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta\right) \underbrace{\Exnb \mathbb{P}_\textbf{X} \left(\left| \left( \hat{P}_n' - P \right)g^{*} \right| \le \frac{\delta }{2} \right)}_{\ge 1/2 \text{ by step 1.} } \\
& \ge \frac{1}{2}\mathbb{P} \left( \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta \right).
\end{aligned}\]
\item
So we can bound the probability of $\left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} > \delta $ by the difference of empirical measures as
\[\begin{aligned}
\mathbb{P} \left( \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} > \delta \right) &\le \mathbb{P} \left( \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta \right) \\
&\le 2 \mathbb{P} \left( \left| \left( \hat{P}_n - P \right)g^{*} \right| > \delta \text{ and } \left| \left( \hat{P}_n' - P \right)g^{*} \right| \le \frac{\delta }{2} \right)&\left( * \right) \\
&\le 2\mathbb{P} \left( \left| \left( \hat{P}_n - \hat{P}_n ' \right)g^{*} \right| > \frac{\delta }{2} \right). &\text{(Triangle)}
\end{aligned}\]
\item Continuing the above, we can use symmetrization to reach our goal
\[\begin{aligned}
\P \left( \left\Vert \hat{P}_n - P \right\Vert _\G > \delta \right) &\le 2 \P\left( \left\Vert \hat{P}_n - \hat{P}_n ' \right\Vert _\G > \frac{\delta }{2} \right)\\
&= 2\mathbb{P} \left( \left\Vert \hat{P}_n ^{\sigma} - \hat{P}_n '^{\sigma} \right\Vert _{\mathcal{G} } > \frac{\delta }{2} \right)\\
&\le 2\left[ \mathbb{P} \left( \left\Vert \hat{P}_n ^{\sigma} \right\Vert _\mathcal{G} > \frac{\delta }{4} \text{ or } \left\Vert \hat{P}_n '^{\sigma} \right\Vert _\mathcal{G} > \frac{\delta }{4} \right) \right] &\text{(Triangle ineq.)} \\
&\le 2\left[ \mathbb{P} \left( \left\Vert \hat{P}_n ^{\sigma} \right\Vert _\mathcal{G} > \frac{\delta }{4} \right) + \mathbb{P} \left( \left\Vert \hat{P}_n '^{\sigma} \right\Vert _\mathcal{G} > \frac{\delta }{4} \right) \right] &\text{(Union bound)} \\
&= 4\mathbb{P} \left( \left\Vert \hat{P}_n ^{\sigma} \right\Vert_\mathcal{G} > \frac{\delta }{4} \right). &\text{(Same distribution)}
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Consider the empirical distribution function defined by $\hat{F}_n (x) = \frac{1}{n}\sum_{i = 1}^{n} 1_{\left( -\infty, x \right]} \left( X_i \right)$. Use symmetrization to show that for any $\delta > 0$ and for large $n$,
\[\begin{aligned}
\P \left( \sup _x \left| \hat{F}_n(x) - F(x) \right| > \delta \right) \le 8\exp \left[ -\frac{n\delta ^{2}}{2^{6}} \right].
\end{aligned}\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item First, to simplify operations, write $\hat{F}_n $ in empirical measure notation as
\[
\hat{F}_n (x) = \frac{1}{n}\sum_{i = 1}^{n} 1_{\left( -\infty, x \right]} \left( X_i \right) = \hat{P}_n 1_{\left( -\infty, x \right]} = \hat{P}_n g_x.
\]
\item We want to be able to use symmetrization, so we need $n$ such that $\var{g} \le \frac{n\delta ^{2}}{8}$. So, we compute the variance
\[
\var{1_{\left( -\infty, x \right]} (X)} = F\left( x \right)\left( 1 - F\left( x \right) \right) \le \frac{1}{4} \le \frac{n\delta ^{2}}{8}.
\]
\item We wish to be able to use Hoeffding's inequality, which requires the $\sup $ to be over finitely many elements. Observe that by definition of $\hat{F}^{\sigma}_n $, the function changes values only at the $X_i$. So, we can write
\[\begin{aligned}
\sup _x \hat{F}_n ^{\sigma }\left( x \right) = \max _{1 \le j \le n + 1} \hat{F}_n ^{\sigma }\left( X_{\left( j \right)} \right) = \max _{1 \le j \le n + 1} \frac{1}{n} \sum_{i = 1}^{n} \sigma _i 1_{\left( -\infty, X_{\left( j \right)} \right]} (X_i)
\end{aligned}\]
where the $+1$ comes from $X_{\left( n + 1 \right)}$ which we take to be infinity.
\item Then by Hoeffding
\[\begin{aligned}
\P \left( \max _{1 \le j \le n + 1} \frac{1}{n} \left| \sum_{}^{} \sigma _i 1_{\left( -\infty, X_{\left( j \right)} \right]} \left( X_i \right) \right| > \sqrt{\frac{2 \left( t + \log \left( n + 1 \right) \right)}{n}} \right) &\le 2\exp \left[ -t \right] \text{ for all $t > 0$} .
\end{aligned}\]
\item To conclude the symmetrization proof, we need to show that with small probability
\[
\left\Vert \hat{P}_n ^{\sigma} \right\Vert_\mathcal{G} = \max _{1 \le j \le n + 1} \frac{1}{n} \left| \sum_{}^{} \sigma _i 1_{\left( -\infty, X_{\left( j \right)} \right]} \left( X_i \right) \right| > \delta /4.
\] so we want
\[
\sqrt{2\frac{ \left( t + \log \left( n + 1 \right) \right)}{n}} \le \delta /4.
\]
\item Set $t = \frac{n\delta ^{2}}{2^{6}}$. Now for very large $n$, we get $\frac{\log \left( n + 1 \right)}{n} \le \frac{\delta ^{2}}{2^{6}} $, which we can plug into the square root to get
\[\begin{aligned}
\sqrt{\frac{2 \left( t + \log \left( n + 1 \right) \right)}{n}} &= \sqrt{2 \left[ \frac{\delta ^{2}}{2^{6} } + \frac{\log \left( n + 1 \right)}{n} \right]} \\
&\stackrel{\text{for large $n$}}{\le } \sqrt{2 \left[ \frac{\delta ^{2}}{2^{6}} + \frac{\delta ^{2}}{2^{6}} \right]} \\
&= \frac{\delta}{4}.
\end{aligned}\]
\item So, combine above steps to conclude
\[\begin{aligned}
&\P \left( \sup _x \left| \hat{F}_n(x) - F(x) \right| > \delta \right) \\
&\le 4\mathbb{P} \left( \sup _x \left| \hat{F}_n ^{\sigma}(x) \right| > \frac{\delta }{4} \right)\\
&= 4\P \left( \max _{1 \le j \le n + 1} \frac{1}{n} \left| \sum_{}^{} \sigma _i 1_{\left( -\infty, X_{\left( j \right)} \right]} \left( X_i \right) \right| > \frac{\delta }{4} \right)\\
&\stackrel{\text{for large $n$} }{\le } 4\P \left( \max _{1 \le j \le n + 1} \frac{1}{n} \left| \sum_{}^{} \sigma _i 1_{\left( -\infty, X_{\left( j \right)} \right]} \left( X_i \right) \right| > \sqrt{2 \left[ \frac{\delta ^{2}}{2^{6} } + \frac{\log \left( n + 1 \right)}{n} \right]} \right)\\
&\le 8 \exp \left[- \frac{n\delta ^{2}}{2^{6}} \right].
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
In context of covering numbers, what does $N_q \left( \delta , \G, \hat{P}_n \right)$ mean?
[/latex]
Back: [latex]
The empirical covering number $N_q \left( \delta , \G, \hat{P}_n \right)$ is the $\delta $-covering number of $\left( \G, \left( \hat{P}_n \left| g \right|^{q} \right)^{1/q} \right).$
[/latex]

Front: [latex]
Consider a function class $\mathcal{G} $.
Sketch a proof to show that if $\mathcal{G} $ is uniformly bounded, i.e. there exists a $K < \infty$ such that
\[
\sup _{g \in \G} \left| g \right| \le K,
\] and
\[
\frac{1}{n}H_1\left( \delta , \G, \hat{P}_n \right) \xrightarrow[n \to \infty]{\P} 0
\] then $\G$ is Glivenko-Cantelli.
[/latex]
Back: [latex]
We will use symmetrization with Hoeffding to ultimately bound
\[
\mathbb{P} \left( \left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} > \delta \right) < \mathbb{P} \left( \left\Vert \hat{P}_n ^{\sigma} \right\Vert > \frac{\delta }{4} \right) \xrightarrow[n \to \infty]{} 0.
\]
\begin{enumerate}[1.]
\item First, we want to bound the infinite supremum by a finite maximum to be able to use Hoeffding. Let $\left\{ g_j \right\}_{j = 1}^{N}$ be a $\frac{\delta }{8}$-covering set of $\left( \G, \hat{P}_n \left| \cdot \right| \right)$ and $N = N_1\left( \delta , \G, \hat{P}_n \right)$. Now we can write
\[\begin{aligned}
\left\Vert \hat{P}_n ^{\sigma } \right\Vert _\mathcal{G} &\le \sup _{g \in \G} \min _{1 \le j \le N} \left| \hat{P}_n ^{\sigma } \left( g - g_j \right) \right| + \max _{1 \le j \le N} \left| \hat{P}_n ^{\sigma }g_j \right| \\
&\le \frac{\delta }{8} + \max _{1 \le j \le N} \left| \hat{P}_n ^{\sigma }g_j \right|.
\end{aligned}\]
\item Now we can bound the probability of the empirical supremum norm
\[\begin{aligned}
\P \left( \left\Vert \hat{P}_n ^{\sigma } \right\Vert_\mathcal{G} > \frac{\delta }{4} \right) &\le \P \left( \frac{\delta }{8} + \max_{1 \le j \le N} \left| \hat{P}_n ^{\sigma }g_j \right| > \frac{\delta }{4} \right)\\
&= \P \left( \max_{1 \le j \le N} \left| \hat{P}_n ^{\sigma }g_j \right| > \frac{\delta }{8} \right).
\end{aligned}\]
\item So, we want to apply Hoeffding. Conditioning on $\textbf{X} $, since $N = N_1\left( \delta , \mathcal{G} , \hat{P}_n \right)$ is random and dependent on it through $\hat{P}_n $, we can write
\[\begin{aligned}
\P_{\textbf{X} } \left( \max _{1 \le j \le N} \left| \hat{P}_n ^{\sigma }g_j \right| > K\sqrt{\frac{2 \left( t + \log \left( 2N \right) \right)}{n}} \right) \le \exp \left[ -t \right].
\end{aligned}\]
\item
To stop conditioning over $\textbf{X}$, we need to choose a $t$ such that
\begin{enumerate}[(a)]
\item if $K\sqrt{\frac{2 \left( t + \log \left( 2N \right) \right)}{n}} \le \frac{\delta }{8}$ then $\exp \left[ -t \right]$ vanishes to 0 as $n \to 0$, and
\item $\P\left( K\sqrt{\frac{2 \left( t + \log \left( 2N \right) \right)}{n}} > \delta /8\right)$ vanishes to 0 as $n \to \infty$.
\end{enumerate}
This is satisfied by $t = \frac{n\delta ^{2}}{2^{8}K^{2}}$. Note, for a bit cleaner notation, \[ K\sqrt{\frac{2 \left( t + \log \left( 2N \right) \right)}{n}} > \delta /8\iff \frac{\log \left( 2N \right)}{n} > \frac{\delta ^{2}}{2^{8}K^{2}}.\]
\item Thus, we conclude with
\[\begin{aligned}
\P \left( \left\Vert \hat{P}_n - P \right\Vert _\G > \delta \right) &\le 4 \mathbb{P} \left( \left\Vert \hat{P}_n^{\sigma} \right\Vert_\mathcal{G} > \frac{\delta }{4} \right) \\
&\le 4\underbrace{\exp \left[ - \frac{n\delta ^{2}}{2^{8}K^{2}} \right]}_{\text{goes to 0} } + 4\underbrace{\P \left( \frac{\log \left( 2N_1 \left( \delta /8, \G, \hat{P}_n \right) \right)}{n} > \frac{\delta ^{2}}{2^{8}K^{2}} \right)}_{\text{$\frac{1}{n}H_1\left( \delta/8 , \mathcal{G} , \hat{P}_n \right) \to 0$ by assumption} }.
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Consider the statement below.
\begin{enumerate}[ ]
\item Let $\mathcal{G} $ be a function class $\mathcal{G} $. If $\mathcal{G} $ is uniformly bounded, i.e. there exists a $K < \infty$ such that
\[
\sup _{g \in \G} \left| g \right| \le K,
\] and
\[
\frac{1}{n}H_1\left( \delta , \G, \hat{P}_n \right) \xrightarrow[n \to \infty]{\P} 0
\] then $\G$ is Glivenko-Cantelli.
\end{enumerate}
Why is it difficult to prove without using symmetrization?
[/latex]
Back: [latex]
Our goal in proving this statement is to somehow bound the term $\left\Vert \hat{P}_n - P \right\Vert _{\mathcal{G} }$ in probability. To do this, we need to turn the supremum over all functions in $\mathcal{G} $ into a finite maximum, so that we can then use Hoeffding.
If we use symmetrization, then we can just write \[\mathbb{P} \left( \left\Vert \hat{P}_n - P \right\Vert_\mathcal{G} > \delta \right) \le \mathbb{P} \left( \left\Vert \hat{P}_n ^{\sigma} \right\Vert _\mathcal{G} > \frac{\delta }{4} \right)\] and rely on a finite covering of $\mathcal{G} $ in the empirical $L^{1} \left( \hat{P}_n \right)$ norm, which we know is finite (and grows slowly with $n$) thanks to the entropy limit condition to write
\[\begin{aligned}
\left\Vert \hat{P}_n ^{\sigma } \right\Vert _\mathcal{G} &\le \sup _{g \in \G} \min _{1 \le j \le N} \left| \hat{P}_n ^{\sigma } \left( g - g_j \right) \right| + \max _{1 \le j \le N} \left| \hat{P}_n ^{\sigma }g_j \right| \\
&\le \frac{\delta }{8} + \max _{1 \le j \le N} \left| \hat{P}_n ^{\sigma }g_j \right|.
\end{aligned}\]
The problem with not using symmetrization is that we do not know the \emph{theoretical} covering of $\mathcal{G} $, which we would need to deal with the $P$ term in $\left\Vert \hat{P}_n - P \right\Vert _\mathcal{G} $. In fact, we would need to somehow combine the theoretical and empirical coverings, which only further complicates things. Using symmetrization, we can rely on just the empirical covering.
[/latex]

Front: [latex]
What does $H_\infty \left( \delta , \G, \hat{P}_n \right)$ mean? How is it different from $H_\infty \left( \delta , \mathcal{G} \right)$?
[/latex]
Back: [latex]
$H_\infty \left( \delta , \G, \hat{P}_n \right)$ is the $\delta $-covering entropy of the metric space on $\mathcal{G} $ induced by the pseudo-norm $\max _{1 \le i \le n} \left| g\left( X_i \right) \right|$.
In contrast, the norm of $H_\infty \left( \delta , \mathcal{G} \right)$ considers the entire domain of the functions in $\mathcal{G} $, not just their values at $X_i$.
[/latex]

Front: [latex]
Show that $H_1 \left( \delta , \G, \hat{P}_n \right) \le H_\infty \left( \delta , \G, \hat{P}_n \right)$ for all $\delta > 0$.
[/latex]
Back: [latex]
\[\begin{aligned}
\hat{P}_n \left| g \right| := \frac{1}{n} \sum_{i = 1}^{n} \left| g\left( X_i \right) \right| \le \max _{1 \le i \le n} \left| g\left( X_i \right) \right|
\end{aligned}\]
[/latex]

Front: [latex]
Consider the class $\mathcal{G} $ of monotone increasing functions from $\mathbb{R} $ to $[0, 1]$ \[\G = \left\{ g : \R \to \left[ 0, 1 \right], g \text{ increasing} \right\}.\]
Show that for all $\delta > 0$, \[H_\infty \left( \delta , \G, \hat{P}_n \right) \le \left( 1 + \frac{1}{\delta } \right) \log \left( n + \frac{1}{\delta } \right),\] and hence that $\mathcal{G} $ is Glivenko-Cantelli.
[/latex]
Back: [latex]
Our goal will be to construct a finite family of discretized $g_j$ that can cover any $g \in \mathcal{G} $ at the $n$ points $X_1, \dots, X_n$, since $H_\infty\left( \cdot, \mathcal{G} , \hat{P}_n \right)$ depends on the pseudo-norm $\max _{1 \le i \le n} \left| g\left( X_i \right) \right|$.
\begin{enumerate}[1.]
\item Let $\delta > 0$. Define the ceiling operator be $\left\lceil z \right\rceil = \min \left\{ m \in \N : z \le m \right\}$. Take any function $g \in \mathcal{G} $.
\item Define the discretized $g_j\left( x \right) = \left\lceil \frac{g(x)}{\delta } \right\rceil \delta .$ This $g_j$ covers $g$ because
\[
\sup _x \left| g_j(x) - g(x) \right| \le \delta.
\]
\begin{center}\includegraphics[width=0.8\textwidth]{ept_piecewise}\end{center}
\item Now we want to count how large the family of $g_j$ is. Each $g_j$ is defined by the $n$ points corresponding to $X_1, \dots, X_n $ and a maximum of $m$ jumps of height $\delta $ it can perform until it reaches 1. So, we can describe the $g_j$ by a sequence of $m + n - 1$ points divided into sets of $n - 1$ and $m$ points. The images illustrate this scheme for a case $n = 8, m = 5$.
\begin{center}\includegraphics[width=0.8\textwidth]{ept_jumps_of_function}\end{center}
\begin{center}\includegraphics[width=0.8\textwidth]{ept_jumps_counts}\end{center}
\item So, the number of possible sequences/piecewise constant functions is $\binom{m + n - 1}{m}$. The bound $m$ on the number of jumps is \[
\#\text{ jumps of $g_j$} \le m = \left\lceil \frac{1}{\delta } \right\rceil \le \frac{1}{\delta } + 1 .
\]
\item So, we can put a bound on $H_\infty\left( \delta , \mathcal{G} , \hat{P}_n \right)$ as
\[\begin{aligned}
H_\infty \left( \delta , \G, \hat{P}_n \right) &= \log N_\infty \left( \delta , \G, \hat{P}_n \right) \\
&\le \log \binom{m + n - 1}{m} \\
&\le m \log \left( m + n - 1 \right) &\text{($\binom{p}{k} \le p^{k}$)} \\
&\le \left( 1 + \frac{1}{\delta } \right) \log \left( \frac{1}{\delta } +n \right).
\end{aligned}\]
\item The above means that $\mathcal{G} $ satisfies the entropy limit condition
\[\begin{aligned}
\frac{1}{n}H_\infty \left( \delta , \G, \hat{P}_n \right)& \le \frac{1}{n} \left( 1 + \frac{1}{\delta } \right) \log \left( \frac{1}{\delta } +n \right) \xrightarrow[n \to \infty]{} 0.
\end{aligned}\]
\item Combined with the fact $\mathcal{G} $ are uniformly bounded by 1, we can conclude that the class $\mathcal{G} $ of monotone increasing functions from $\mathbb{R}$ to $[0, 1]$ is Glivenko-Cantelli.
\end{enumerate}
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $. Show that if
\begin{enumerate}[(a)]
\item there exists an envelope function $G$ for $\mathcal{G} $ which is $P$-integrable, i.e. $PG < \infty$, and
\item $\frac{1}{n}H_1 \left( \delta , \mathcal{G} , \hat{P}_n \right) \xrightarrow[n \to \infty]{\mathbb{P} } 0$ for all $\delta > 0$,
\end{enumerate}
then $\mathcal{G} $ is Glivenko-Cantelli.
[/latex]
Back: [latex]
It holds that for all $g \in \mathcal{G} $ and for any $K > 0$
\[
\left| \left( \hat{P}_n - P \right)g \right| \le \underbrace{\left| \left( \hat{P}_n - P \right)g 1_{\left\{ G \le K \right\}} \right|}_{:= (i)} + \underbrace{\left| \left( \hat{P}_n - P \right)g1_{\left\{ G > K \right\}} \right|}_{:= (ii)}.
\]
Define the helper classes of functions
\[\begin{aligned}
\G_{\le K} &:= \left\{ g 1_{\left\{ \G \le K \right\}} : g \in \G \right\}\\
\G_{> K} &:= \left\{ g 1_{\left\{ \G > K \right\}} : g \in \G \right\}.
\end{aligned}\]
\begin{enumerate}[(i)]
\item The class $\G_{\le K}$ is uniformly bounded by $K$, and satisfies $H_1 \left( \cdot, \mathcal{G} _{\le K}, \hat{P}_n \right) \le H_1 \left( \cdot, \mathcal{G} , \hat{P}_n \right)$ which we assumed to tend to 0 as $n \to \infty$. So, we can conclude it is Glivenko-Cantelli, giving us
\[
\left\Vert \hat{P}_n - P \right\Vert _{\mathcal{G}_{\le K} } \xrightarrow[n \to \infty]{\mathbb{P} } 0.
\]
\item This term we can bound by terms that go to 0 as $n \to \infty$ and $K \to \infty$
\[\begin{aligned}
\left\Vert \hat{P}_n - P \right\Vert _{\mathcal{G} _{>K}}&= \sup _{g \in \G}\left| \left( \hat{P}_n - P \right) g 1_{\left\{ G > K \right\}} \right|\\
&\le \left( \hat{P}_n + P \right)G 1_{\left\{ G > K \right\}}\\
&=\underbrace{\left( \hat{P}_n - P \right)G1_{\left\{ G > K \right\}}}_{\xrightarrow[n \to \infty]{\mathbb{P} } 0 \text{ } \forall K \text{ by standard LLN} } + \underbrace{2PG1_{\left\{ G > K \right\}}}_{\xrightarrow[K \to \infty]{} 0 \text{ by integrability} }.
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Let $\mathcal{D} $ be a collection of subsets of $\mathcal{X} $, and let $ \xi_1, \dots, \xi_n $ be $n$ points in $\mathcal{X} $. What does $\Delta^{\mathcal{D}}\left( \xi_1 , \dots , \xi_n \right)$ mean? How is it related to the covering number $N_\infty$?
[/latex]
Back: [latex]
$\Delta^{\mathcal{D}}\left( \xi_1 , \dots , \xi_n \right)$ is the intersection size between class $\mathcal{D} $ and subset $\left\{ \xi_{1} , \dots , \xi_{n} \right\}$, defined as
\[\begin{aligned}
\Delta^{\mathcal{D}}\left( \xi_1 , \dots , \xi_n \right) &= \#\left( \left\{ D \cap \left\{ \xi_1 , \dots , \xi_n \right\} : D \in \mathcal{D} \right\} \right)\\
&=\text{ the number of subsets of $\left\{ \xi_{1} , \dots , \xi_{n} \right\}$ that $\mathcal{D} $ distinguishes.}
\end{aligned}\]
We can define it alternatively using covering number notation. Define a function class of indicator functions $\G := \left\{ 1_{D} : D \in \D \right\} \triangleq \D$, and note that
\[\begin{aligned}
\max _{1 \le i \le n} \left| 1_{D_1}\left( \xi_i \right) - 1_{D_2} \left( \xi_i \right) \right| =
\begin{cases}
1 &\text{ if }\exists_{} \xi_i \in D_1 \Delta D_2 \\
0 &\text{ otherwise.}
\end{cases}
\end{aligned}\] so we can write
\[
\Delta^{\mathcal{D} }\left( X_1, \dots, X_n \right) = N_\infty \left( \delta , \D, \hat{P}_n \right) \text{ when $0 < \delta < 1$} .
\]
[/latex]

Front: [latex]
Let $\mathcal{D} $ be a family of left half-intervals of $\mathbb{R} $
\[
\mathcal{D} = \left\{ \left( -\infty, x \right] : x \in \mathbb{R} \right\},
\] and let $ \xi_1, \dots, \xi_n $ be $n$ points in $\mathbb{R} $. What is an upper bound for $\Delta^{\mathcal{D} } \left( \xi_{1} , \dots , \xi_{n} \right)$?
[/latex]
Back: [latex]
\begin{center}\includegraphics[width=0.8\textwidth]{ept_vc_dimension_example}\end{center} in the image above, both of the red and both of the green lines pick out the same points, so they're considered the same.
\[
\Delta^{\mathcal{D} } \left( \xi_1, \dots, \xi_n \right) \le n + 1.
\]
[/latex]

Front: [latex]
Let $\mathcal{D} $ be the collection of all subsets of $\mathcal{X} $ and let $ \xi_1, \dots, \xi_n $ be $n$ points in $\mathcal{X} $. What is an upper bound for $\Delta^{\mathcal{D} } \left( \xi_{1} , \dots , \xi_{n} \right)$?
[/latex]
Back: [latex]
Since $\mathcal{D} $ can distinguish all subsets of $\xi_{1} , \dots , \xi_{n} $, we have
\[
\Delta^{\mathcal{D} } \left( \xi_{1} , \dots , \xi_{n} \right) = 2^{n}.
\]
[/latex]

Front: [latex]
Show that if
\[
\frac{1}{n} \log \Delta^{\D} \left( X_1 , \dots , X_n \right) \xrightarrow[n \to \infty]{\P} 0
\] then
\[
\sup _{D \in \D} \left| \hat{P}_n(D) - P(D) \right| \xrightarrow[n \to \infty]{\P} 0.
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item Consider the family of indicator functions $\mathcal{G} = \left\{ 1_D : D \in \mathcal{D} \right\}$. Now note that for $\delta \in \left( 0, 1 \right)$ we can write
\[\begin{aligned}
\Delta^{\mathcal{D} } \left( X_{1} , \dots , X_{n} \right) &= N_\infty\left( \delta , \mathcal{G}, \hat{P}_n \right),\text{ and } \\
\log \Delta^{\mathcal{D} } \left( X_{1} , \dots , X_{n} \right) &= H_\infty\left( \delta , \mathcal{G}, \hat{P}_n \right).
\end{aligned}\]
\item Now note that
\[\begin{aligned}
\frac{1}{n} H_1\left( \delta , \mathcal{G} , \hat{P}_n \right) &\le \frac{1}{n} H_\infty\left( \delta , \mathcal{G} , \hat{P}_n \right) &\text{(Property of $H_q$)} \\
&= \frac{1}{n} \log \Delta^{\mathcal{D} } \left( X_{1} , \dots , X_{n} \right)\\
&\xrightarrow[n \to \infty]{\mathbb{P} } 0. &\text{(Assumption)}
\end{aligned}\]
\item We can combine the above bound on $L^{1} $ entropy with the fact that $\mathcal{G} $ is uniformly bounded to conclude that it is Glivenko-Cantelli.
\end{enumerate}
[/latex]

Front: [latex]
For a class of subsets $\mathcal{D} $ of $\mathcal{X} $, what does $m^{\mathcal{D} }(n)$ mean?
[/latex]
Back: [latex]
$m^{\mathcal{D} }(n)$ is the shattering number defined as
\[
m^{\mathcal{D} }(n) = \sup \left\{ \Delta^{\mathcal{D} } \left( \xi_{1} , \dots , \xi_{n} \right) : \xi_{1} , \dots , \xi_{n} \in \mathcal{X} \right\},
\] which in other words means the supremum intersection size between the class $\mathcal{D} $ and a subset of size $n$ of $\mathcal{X} $.
[/latex]

Front: [latex]
What does it mean for a class of sets $\mathcal{D} $ to be a Vapnik-Chervonenkis class?
[/latex]
Back: [latex]
Let $m^{\mathcal{D} }(n)$ be the shattering number of $\mathcal{D} $
\[
m^{\mathcal{D} }(n) = \sup \left\{ \Delta^{\mathcal{D} } \left( \xi_{1} , \dots , \xi_{n} \right) : \xi_{1} , \dots , \xi_{n} \in \mathcal{X} \right\}.
\]
We say that $\mathcal{D} $ is a Vapnik-Chervonenkis (VC) class if for certain constants $c$ and $V$, and for all $n$,
\[
m^{\mathcal{D} }(n) \le cn^{V},
\] i.e. if $m^{\mathcal{D} }(n)$ does not grow faster than a polynomial in $n$.
[/latex]

Front: [latex]
How is a class of sets being Vapnik-Chervonenkis relate to it being Glivenko-Cantelli?
[/latex]
Back: [latex]
For sets $\text{VC} \implies \text{GC} $, but not the other way.
[/latex]

Front: [latex]
Let $\X = \R$ and consider the class of half intervals $\left\{ 1_{\left( -\infty, x \right]} : x \in \R \right\} = \D$. What is the shattering number $m^{\mathcal{D}}\left(n\right) $?
[/latex]
Back: [latex]
For the class of half intervals,
\[
m^{\mathcal{D}}\left(n\right) = n + 1.
\]
\begin{center}\includegraphics[width=0.8\textwidth]{ept_shattering_hinterval}\end{center}
[/latex]

Front: [latex]
Let $\X = \R^{d}$ and consider the class of $d$-dimensional half intervals $\D = \left\{ 1_{\left( -\infty, x \right]} : x \in \R^{d} \right\}$. What is the shattering number $m^{\mathcal{D}}\left(n\right) $?
[/latex]
Back: [latex]
For $d$-dimensional half intervals the shattering number is
\[
m^{\D} \left( n \right) \le \left( n + 1 \right)^{d}.
\]
\begin{center}\includegraphics[width=0.8\textwidth]{ept_shattering_multidimhinterval.png}\end{center}
[/latex]

Front: [latex]
Let $\textbf{X} _{1} , \dots , \textbf{X} _{n}$ be random vectors in $\mathbb{R} ^{d}$. How can you show that the $d$-dimensional empirical distribution function $\hat{F}_n $ converges in probability to the true distribution function $F$ as $n \to \infty$ using properties of Vapnik-Chervonenkis classes?
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item
We can write $"\textbf{X} \le \textbf{Y} "$ for vectors as $\textbf{X} \in \left( -\infty, \textbf{Y} \right]$. Note that $\hat{F }_n $ is then defined as
\[
\hat{F}_n \left( \textbf{x} \right) = \hat{P}_n \left( \left\{ \textbf{X} _i \in \left( -\infty, \textbf{x} \right] : i \in \left\{ 1, \dots, n \right\} \right\} \right)
\] so to show that it converges in probability to $F$, we want the family of $d$-dimensional half-intervals to be Glivenko-Cantelli.
\item We know that if a class of sets is Vapnik-Chervonenkis, then it is also Glivenko-Cantelli. So, we will show that the family of $d$-dimensional half-intervals is Vapnik-Chervonenkis.
\item We know that for $d$-dimensional half intervals the shattering number is
\[
m^{\D} \left( n \right) \le \left( n + 1 \right)^{d},
\] which grows polynomially in $n$, and thus we know that $d$-dimensional half intervals are Vapnik-Chervonenkis.
\begin{center}\includegraphics[width=0.8\textwidth]{ept_shattering_multidimhinterval.png}\end{center}
\end{enumerate}
[/latex]

Front: [latex]
Let $\X = \R^{d}$ and consider the class $\mathcal{D} $ of all half-spaces in $\mathbb{R} ^{d}$. Show that $\mathcal{D} $ is Vapnik-Chervonenkis.
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item To show $\mathcal{D} $ is Vapnik-Chervonenkis, we need its shattering number to grow at most polynomially in $n$. So, we will try to find an upper bound.
\item Note that in $d$-dimensions, a half-space is identified by $d$ points, giving us a factor of $\binom{n}{d} $. Then, each of these $d$ points can be either "above" or "below" the half-space, giving another factor of $2^{d}$.
\begin{center}\includegraphics[width=0.8\textwidth]{ept_shattering_halfspace}\end{center}
\item So, we can bound the shattering number as
\[
m^{\mathcal{D}}\left(n\right) = 2^{d}\binom{n}{d} \le 2^{d}n^{d}
\] showing that indeed $\mathcal{D} $ is Vapnik-Chervonenkis.
\end{enumerate}
[/latex]

Front: [latex]
What are the closure properties of Vapnik-Chervonenkis classes?
[/latex]
Back: [latex]
Vapnik-Chervonenkis classes $\mathcal{D} $, $\mathcal{D} _1$, $\mathcal{D} _2$ are closed under
\begin{enumerate}[(1)]
\item complements $\mathcal{D}^{C} = \left\{ D^{C} : D \in \mathcal{D} \right\}$,
\item intersections $\mathcal{D} _1 \cap \mathcal{D} _2 = \left\{ D_1 \cap D_2 : D_1 \in \mathcal{D} _1, D_2 \in \mathcal{D} _2 \right\}$, and
\item unions $\mathcal{D} _1 \cup \mathcal{D} _2 = \left\{ D_1 \cup D_2 : D_1 \in \mathcal{D} _1, D_2 \in \mathcal{D} _2 \right\}$.
\end{enumerate}
[/latex]

Front: [latex]
Is the class of ellipsoids Vapnik-Chervonenkis? Explain why.
[/latex]
Back: [latex]
\[
\left\{ \left\{ \left( x, y \right) : \sigma _{11}x^{2} + \sigma _{12}xy + \sigma _{22}y^{2} \le t \right\}, \left( \sigma _{11}, \sigma _{12}, \sigma _{22}, t \right) \subset \R^{4} \right\}
\] consider $x^{2} = x_1$, $xy = x_2$, $y^{2} = x_3$; in these variables, it's just a half-space, thus we can tell it's VC.
[/latex]

Front: [latex]
Is the class of all convex subsets of $\left[ 0, 1 \right]^{2}$ Vapnik-Chervonenkis? Explain why.
[/latex]
Back: [latex]
The class of all convex subsets of $\left[ 0, 1 \right]^{2}$ is not Vapnik-Chervonenkis.
Consider a set $D$ of equidistant points around a circle. A convex set can pick out any subset of $D$, so the family of convex sets cannot be Vapnik-Chervonenkis.
\begin{center}\includegraphics[width=0.8\textwidth]{ept_vapnik_chervonenkis_convex_sets}\end{center}
[/latex]

Front: [latex]
Is the class of all monotone subsets of $\mathbb{R} ^{2}$ Vapnik-Chervonenkis? Explain why.
Recall that $D$ is monotone if for all $\left( x, y \right) \in D$ then $\left( \tilde{x}, \tilde{y} \right) \in D$ for all $\tilde{x} \le x, \tilde{y} \le y$.
[/latex]
Back: [latex]
The class of all monotone subsets of $\mathbb{R} ^{2}$ is not Vapnik-Chervonenkis.
Consider a set of points on a line. The family of monotone sets can pick out any subset of these points.
\begin{center}\includegraphics[width=0.8\textwidth]{ept_vapnik_chervonenkis_monotone_sets}\end{center}
[/latex]

Front: [latex]
Give an example of a class of sets which is Glivenko-Cantelli but not Vapnik-Chervonenkis.
[/latex]
Back: [latex]
The class of all convex subsets of $\left[ 0, 1 \right]^{2}$ is Glivenko-Cantelli if $P$ is uniform, but it is not Vapnik-Chervonenkis.
[/latex]

Front: [latex]
What is the Vapnik-Chervonenkis dimension of a class of sets $\mathcal{D} $?
[/latex]
Back: [latex]
The VC dimension of $\D$ is
\[
V\left( \mathcal{D} \right) := \max \left\{ n : m^{\D}\left( n \right)=2^{n} \right\}.
\] This is the maximum size of a set where $\D$ can pick out all subsets.
[/latex]

Front: [latex]
What are the packing number and capacity of a metric space $\left( S, d \right)$?
[/latex]
Back: [latex]
For $\delta > 0$, the $\delta $-packing number is the maximum number of points in this space which are at distance at most $\delta $, i.e. largest $N$ such that there exists $\left\{ s_j \right\}_{j = 1}^{N} \in S$ with $d \left( s_j, s_k \right) > \delta $ for all $j \neq k$. Such a max set is called a $\delta $-packing set.
\[
N = D\left( \delta , S, d \right) := \text{$\delta $-packing number}.
\]
\[
\log D\left( \delta , S, d \right) = \text{ capacity of $\left( S, d \right)$} .
\]
"How many oranges can you put in the box?" \begin{center}\includegraphics[width=0.8\textwidth]{ept_how_many_oranges_in_box}\end{center}
[/latex]

Front: [latex]
What are the connections between the covering and packing numbers of a metric space $\left( S, d \right)$?
[/latex]
Back: [latex]
For all $\delta > 0$,
\begin{enumerate}[(i)]
\item $N\left( \delta , S, d \right) \le D\left( \delta , S, d \right)$,
\item $D\left( \delta , S, d \right) \le N\left( \delta /2, S, d \right)$.
\end{enumerate}
[/latex]

Front: [latex]
Consider a metric space $\left( S, d \right)$. Show that for all $\delta > 0$,
\begin{enumerate}[(i)]
\item $N\left( \delta , S, d \right) \le D\left( \delta , S, d \right)$,
\item $D\left( \delta , S, d \right) \le N\left( \delta /2, S, d \right)$.
\end{enumerate}
[/latex]
Back: [latex]
Let $s_{1} , \dots , s_{N} $ in $S$ with $d\left( s_k, s_j \right) > \delta $ for all $k \neq j$. Let it be a maximal such set.
\begin{enumerate}[(i)]
\item Then for any $s \in S$ there is a $j \in \left\{ 1, \dots, N \right\}$ such that $d\left( s, s_j \right) \le \delta $, since otherwise we could add $s$ to the packing.
\item Since $\left\{ s_1, \dots, s_N \right\} \subset S$,
\[
N\left( \delta /2, S, d \right) \ge N\left( \delta /2, \left\{ s_1, \dots, s_N \right\}, d \right).
\]
But
\[
N\left( \delta /2, \left\{ s_1, \dots, s_N \right\}, d \right) = N.
\]
\end{enumerate}
[/latex]

Front: [latex]
What is the graph (subgraph) of a function $g: \mathcal{X} \to \mathbb{R} $?
[/latex]
Back: [latex]
The graph (subgraph) of $g$ is
\[
\text{graph} \left( g \right) = \left\{ \left( x, t \right) \in \X \times \R : g(x) \ge t \right\}.
\]
[/latex]

Front: [latex]
What does it mean for a class of functions $\mathcal{G} $ to be called Vapnik-Chervonenkis?
[/latex]
Back: [latex]
A class of functions $\mathcal{G} $ is called Vapnik-Chervonenkis if the collection of its graphs $\left\{ \text{graph} \left( g \right) : g \in \G \right\}$ is Vapnik-Chervonenkis.
Recall that the graph of a function $g$ is defined as
\[
\text{graph} \left( g \right) = \left\{ \left( x, t \right) \in \X \times \R : g(x) \ge t \right\}.
\]
[/latex]

Front: [latex]
Show that the family $\mathcal{G} $ of parametrized exponential functions
\[
\G = \left\{ \left\{ g(x) = e^{\theta x} \right\}, \theta \in \R \right\}
\] is Vapnik-Chervonenkis.
[/latex]
Back: [latex]
We can rewrite it to show that its graph is equivalent to a half-space. See that
\[
\text{graph} \left( x \mapsto e^{\theta x} \right) = \left\{ \left( x, t \right): e^{\theta x}\ge t \right\} = \left\{ \left( x, t \right) : \theta x \ge \log t \right\}
\] so it is Vapnik-Chervonenkis.
[/latex]

Front: [latex]
Suppose $C$ and $V$ are some constants, and take $N$ to be a strictly positive integer. Show that
\[
N \le C \left( \log N \right)^{V} \implies N \le C \log ^{V}\left( C^{2} \left( 2V \right)^{2V} \right).
\] [/latex]
Back: [latex]
\[\begin{aligned}
\left( \log N \right)^{V} &= \left( 2V \log N^{\frac{1}{2V}} \right)^{V}\\
&\le \left( 2V \right)^{V}N^{\frac{1}{2}} &(\log x \le x \text{ if } x \ge 1)
\end{aligned}\] which implies that
\[\begin{aligned}
N &\le C \left( 2V \right)^{V}N^{\frac{1}{2}}\\
N &\le C^{2}\left( 2V \right)^{2V}\\
N &\le C \log \left( C^{2}\left( 2V \right)^{2V} \right).
\end{aligned}\]
[/latex]

Front: [latex]
Show, for some numbers $N, n \in \mathbb{N} $, $\delta \in \left( 0, 1 \right)$ that
\[
\binom{N}{2} \left( 1 - \frac{\delta }{2} \right)^{n} \le 1 \text{ if } \frac{n\delta }{2} \ge 2\log N.
\]
[/latex]
Back: [latex]
\[\begin{aligned}
\binom{N}{2} \left( 1 - \frac{\delta }{2} \right)^{n} &\le \frac{N^{2}}{2} \left( 1 - \frac{\delta }{2} \right)^{n} & \left( \binom{N}{2} = \frac{N!}{2!\left( N - 2 \right)!} \le \frac{N^{2}}{2} \right) \\
&\le \frac{N^{2}}{2} \exp \left[ - \frac{n\delta }{2} \right] & \left( \left( 1 - x \right) \le e^{-x} \right)\\
&= \frac{1}{2}\exp \left[ 2 \log N - \frac{n\delta }{2} \right]\\
&\le \frac{1}{2} \text{ for $\frac{n\delta }{2} \ge 2\log N$} .
\end{aligned}\]
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $ with an envelope function $G$. Let $Q$ be an arbitrary probability measure on $\X$. Suppose $\G$ is Vapnik-Chervonenkis, so that its shattering number satisfies $m^{\D}\left( n \right) \le cn^{V}$ where $\D = \left\{ \text{graph} (g) : g \in \G \right\}$.
Show that then there exists a constant $A = A\left( c, V \right)$ such that for all $\delta >0$,
\[
N_1\left( \delta Q G, \G, Q \right) \le A \left( \frac{1}{\delta } \right)^{V}\log ^{V}\left( \frac{1}{\delta } \right)
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item
Without loss of generality, assume $QG = 1$.
\item
Let $\left\{ g_1, \dots, g_N \right\}$ be a (max) $\delta $-packing set, so that for all $j \neq k$
\[\begin{aligned}
Q \left| g_j - g_k \right| > \delta.
\end{aligned}\] Note that this means $N = D_1\left( \delta , \G, Q \right)$.
\item
Let $S$ have distribution $P \left( S \in A \right) = \int_{A}^{} GdQ$ for all measurable $A \subset \X$. In other words, $S$ is a random variable with density $G$. Given $S = s$ let $T$ be uniform on the "enveloped" set $\left[ -G\left( s \right), G\left( s \right) \right]$.
\item Now we can write the conditional probability of $T$ being in an interval bounded by $g_k(S), g_j(S)$
\[\begin{aligned}
\mathbb{P} \left( T \in \left[ g_k(s), g_j(s) \right] | S = s \right) &=\frac{\left| g_k(s) - g_j(s) \right|}{2G(s)}\\
\mathbb{P} \left( T \in \left[ g_k(S), g_j(S) \right] \right) &= \int_{}^{} \frac{\left| g_j - g_k \right|}{2G} GdQ > \frac{\delta }{2}.
\end{aligned}\]
\begin{center}\includegraphics[width=0.8\textwidth]{ept_proof_vc_envelope}\end{center}
\item Let $\left( T_1, S_1, \right), \dots, \left( T_n, S_n \right)$ be i.i.d. $\sim \left( T, S \right)$. Now for some $g_j, g_k$ and for all $T_i$ consider the complement of the above probability
\[\begin{aligned}
\P\left( T_i \cancel{\in} \left[ g_k(S_i), g_j(S_i) \right] \text{ for all $i$} \right) &\le \left( 1 - \frac{\delta }{2} \right)^{n} \end{aligned}\]
and, union bounding over the $g_j, g_k$,
\[\begin{aligned}
\P\left( \exists_{} \left( j, k \right) : T_i \cancel{\in} \left[ g_k (S_i), g_j(S_i) \right] \text{ for all $i$} \right) &\le \binom{N}{2} \left( 1 - \frac{\delta }{2} \right)^{n}\\
&\le \frac{N^{2}}{2} \exp \left[ - \frac{n\delta }{2} \right] &\left( \left( 1 - x \right) \le e^{-x} \right) \\
&= \frac{1}{2}\exp \left[ 2 \log N - \frac{n\delta }{2} \right]\\
&\le \frac{1}{2} \text{ for $\frac{n\delta }{2} \ge 2\log N$} .
\end{aligned}\]
\item Now take the complement
\[\begin{aligned}
\P \left( \forall \left( j, k \right) : T_i \in \left[ g_k(S_i), g_j(S_i) \right]\exists i \right) \ge \frac{1}{2} > 0.
\end{aligned}\]
This event means that in each pair $\left( g_j, g_k \right)$, one function maps to one set from $\mathcal{D} \cap \left\{ \left( T_i, S_i \right)\right\}_{i = 1}^{n}$, the other to another, meaning that\[
\Delta^{\D}\left( \left( T_1, S_1, \right), \dots, \left( T_n, S_n \right) \right) = \left| \mathcal{D} \cap \left\{ \left( T_i, S_i \right) \right\}_{i = 1}^{n} \right| \ge N = D_1\left( \delta , \G, Q \right).
\]
\item If $2\log N \le \frac{n\delta }{2}$ then the above event has positive probability and can occur, so $N \le cn^{V}$ by assumption that $\mathcal{G} $ is Vapnik-Chervonenkis.
\item We want to find the smallest possible value of $n$ such that the above holds, giving us \[n = \left\lceil \frac{4\log N}{\delta } \right\rceil.\] So for such a value of $n$, with positive probability, for any $j \neq k$, some of the $T_i$ fall in between the graphs of $g_j$ and $g_k$ .
\item Inserting the $n$ into the inequality, we get
\[\begin{aligned}
N &\le c \left\lceil \frac{4\log N}{\delta } \right\rceil^{V} &\le c\left( \frac{8\log N}{\delta } \right)^{V} = \frac{c8^{V}}{\delta^{V} } \log ^{V}N
\end{aligned}\] so by, a technical lemma we can transport $N$ onto the left side to find, for some $A = A(c, V)$,
\[\begin{aligned}
N &\le \frac{c8^{V}}{\delta ^{V}} \log ^{V} \left( c^{2}\left( \frac{8}{\delta } \right)^{2V} \left( 2V \right)^{2V} \right) = A\left( \frac{1}{\delta } \right)^{V} \log ^{V}\left( \frac{1}{\delta } \right).
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Show that if a class of functions $\mathcal{G} $
\begin{enumerate}[(1)]
\item has an envelope $G$ with $PG < \infty$, and
\item is Vapnik-Chervonenkis
\end{enumerate} then its covering number grows slower than exponentially, i.e. for any $\delta > 0$
\[
\frac{1}{n}\log N_1\left( \delta , \mathcal{G} , \hat{P}_n \right) = \frac{1}{n} H_1\left( \delta , \mathcal{G} , \hat{P}_n \right)\xrightarrow[n \to \infty]{\mathbb{P} } 0.
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item We know that if the given conditions hold, then there is a constant $A$ such that for any $\delta > 0$,
\[\begin{aligned}
N_1 \left( \delta \hat{P}_n G, G, \hat{P}_n \right) &\le N\left( \delta \right) &=: A\left( \frac{1}{\delta } \right)^{V} \log ^{V}\left( \frac{1}{\delta } \right) \\
N_1 \left( \delta, G, \hat{P}_n \right) &\le N\left( \frac{\delta }{\hat{P}_n G} \right).
\end{aligned}\]
\item Now note that the event $N_1\left( \delta , \G, \hat{P}_n \right) > N\left( \frac{\delta }{2PG} \right)$ implies
\[\begin{aligned}
N\left( \frac{\delta }{\hat{P}_n g} \right) &> N\left( \frac{\delta }{2PG} \right)\\
\implies \hat{P}_n G &> 2PG. &\text{($N$ is decreasing)}
\end{aligned}\]
\item Now, since $PG < \infty$, we know by the law of large numbers that
\[
\left( \hat{P}_n - P \right)G \xrightarrow[n \to \infty]{\text{a.s.} } 0
\] so
\[\begin{aligned}
\P\left( N_1\left( \delta , \G, \hat{P}_n \right) > N\left( \frac{\delta }{2PG} \right) \right) &\le \P\left( \hat{P}_n G > 2PG \right) \\
&= \mathbb{P} \left( \left( \hat{P}_n - P \right)G > PG\right)\\
&\xrightarrow[n \to \infty]{} 0.
\end{aligned}\]
\item So, $N_1\left( \delta , \mathcal{G} , \hat{P}_n \right)$ is bounded in probability, so
\[
\frac{1}{n} H_1\left( \delta , \mathcal{G} , \hat{P}_n \right) \xrightarrow[n \to \infty]{\mathbb{P} } 0.
\]
\end{enumerate}
[/latex]

Front: [latex]
Show that if a class of functions $\mathcal{G} $
\begin{enumerate}[(1)]
\item has an envelope $G$ with $PG < \infty$, and
\item is Vapnik-Chervonenkis
\end{enumerate} then $\mathcal{G} $ is Glivenko-Cantelli.
[/latex]
Back: [latex]
We will show that $\mathcal{G} $ is Glivenko-Cantelli by bounding its covering number and entropy by a constant, hence showing that $\frac{1}{n} H_1\left( \delta , \mathcal{G} , \hat{P}_n \right) \xrightarrow[n \to \infty]{\mathbb{P} } 0$.
\begin{enumerate}[1.]
\item We know that if the given conditions hold, then there is a constant $A$ such that for any $\delta > 0$,
\[\begin{aligned}
N_1 \left( \delta \hat{P}_n G, G, \hat{P}_n \right) &\le N\left( \delta \right) &=: A\left( \frac{1}{\delta } \right)^{V} \log ^{V}\left( \frac{1}{\delta } \right) \\
N_1 \left( \delta, G, \hat{P}_n \right) &\le N\left( \frac{\delta }{\hat{P}_n G} \right).
\end{aligned}\]
\item Now note that the event $N_1\left( \delta , \G, \hat{P}_n \right) > N\left( \frac{\delta }{2PG} \right)$ implies
\[\begin{aligned}
N\left( \frac{\delta }{\hat{P}_n g} \right) &> N\left( \frac{\delta }{2PG} \right)\\
\implies \hat{P}_n G &> 2PG. &\text{($N$ is decreasing)}
\end{aligned}\]
\item Now, since $PG < \infty$, we know by the law of large numbers that
\[
\left( \hat{P}_n - P \right)G \xrightarrow[n \to \infty]{\text{a.s.} } 0
\] so
\[\begin{aligned}
\P\left( N_1\left( \delta , \G, \hat{P}_n \right) > N\left( \frac{\delta }{2PG} \right) \right) &\le \P\left( \hat{P}_n G > 2PG \right) \\
&= \mathbb{P} \left( \left( \hat{P}_n - P \right)G > PG\right)\\
&\xrightarrow[n \to \infty]{} 0.
\end{aligned}\]
\item So, $N_1\left( \delta , \mathcal{G} , \hat{P}_n \right)$ converges in probability to some constant, so
\[
\frac{1}{n} H_1\left( \delta , \mathcal{G} , \hat{P}_n \right) \xrightarrow[n \to \infty]{\mathbb{P} } 0
\] and so $\mathcal{G} $ is Glivenko-Cantelli.
\end{enumerate}
[/latex]

Front: [latex]
Let $X_1, \dots, X_n $ be i.i.d. copies of $X \in \mathcal{X} $. Let $\Theta $ be a parameter space, and consider the problem of estimating a true parameter $\theta $. What are
\begin{enumerate}[1.]
\item $\gamma _\theta $,
\item $R(\theta )$,
\item $\theta _0$,
\item $\hat{\theta }_n $.
\end{enumerate}
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item $\gamma _\theta : \mathcal{X} \to \mathbb{R} $ is the loss function,
\item $R(\theta ) := P\gamma _\theta $ is the true risk,
\item $\theta _0 := \arg \min _{\theta \in \Theta } R(\theta )$ is the optimization target/true minimzer,
\item $\hat{\theta }_n := \arg \min _{\theta \in \Theta } \hat{R}_n \left( \theta \right)$ is the M-estimator?
\end{enumerate}
[/latex]

Front: [latex]
Let $X_1, \dots, X_n $ be i.i.d. copies of $X \in \mathcal{X} $. Let $\Theta $ be a parameter space, and consider the problem of estimating a true parameter $\theta $. Explain what the implication below means, and prove it.
\[
\sup _\theta \left| \hat{R}_n(\theta ) - R(\theta ) \right| \xrightarrow[n \to \infty]{\P} 0 \implies R\left( \hat{\theta }_n \right) \xrightarrow[n \to \infty]{\P} R(\theta_0 ).
\]
[/latex]
Back: [latex]
The implication states that if the loss function class $\left\{ \gamma _\theta : \theta \in \Theta \right\}$ is Glivenko-Cantelli, then the risk of the M-estimator will converge in probability to the risk of the true minimizer.
\begin{center}\includegraphics[width=0.8\textwidth]{ept_empirical_risk_convergence}\end{center}
It is true, because
\[\begin{aligned}
0 &\le R\left( \hat{\theta }_n \right) - R\left( \theta _0 \right) \\
&= \underbrace{\left[ \hat{R}_n(\theta _0) - R(\theta _0) \right] - \left[ \hat{R}_n(\hat{\theta }_n ) - R(\hat{\theta }_n ) \right]}_{\xrightarrow[n \to \infty]{\P} 0 \text{ by Glivenko-Cantelli}} + \underbrace{\hat{R}_n \left( \hat{\theta }_n \right) - \hat{R}_n \left( \theta _0 \right)}_{\le 0}.
\end{aligned}\]
[/latex]

Front: [latex]
Consider the problem of classification. Let the data be $\left( X_1, Y_1 \right), \dots, \left( X_n, Y_n \right)$, with $X \in \mathbb{R} ^{d}, Y \in \left\{ 0, 1 \right\}.$ Take the misclassification loss function
\[
\gamma_D\left( x, y \right) := \left| 1_D \left( x \right) - y \right|
\] and let the hypothesis class $\mathcal{D} $ be the class of all half-spaces. Show that
\[
R\left( \hat{D}_n \right) \xrightarrow[n \to \infty]{\mathbb{P} } R\left( D_0 \right).
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item The class of graphs of $\gamma _\theta $ $\left\{ \text{graph} \left( \gamma _\theta \right):\theta \in \Theta \right\}$ is the class of all half-spaces, which we know is Vapnik-Chervonenkis. Furthermore, $\gamma _\theta $ is bounded by 1. So, we know $\gamma _\theta $ is Glivenko-Cantelli, giving us
\[
\sup _{D \in \mathcal{D} } \left| \hat{R}_n \left( D \right) - R\left( D \right)\right| = \sup _{D \in \mathcal{D} } \left| \left( \hat{P}_n - P \right)\gamma _D \right| \xrightarrow[n \to \infty]{\mathbb{P} } 0.
\]
\item We know from a lemma that if the class of loss functions is Glivenko-Cantelli, then risk of the M-estimator converges to the risk of the true minimizer
\[
R\left( \hat{D}_n \right) \xrightarrow[n \to \infty]{\mathbb{P} } R\left( D_0 \right).
\]
\end{enumerate}
[/latex]

Front: [latex]
Consider the problem of 1-bit compressed sensing. Let $X_i \sim \mathcal{N} \left( 0, I \right)$, $Y_i \in \left\{ 0, 1 \right\}$. The class of loss functions is $\mathcal{G} = \left\{ \gamma _\theta \left( x, y \right) = -yx\theta : \left\Vert \theta \right\Vert _2 \le 1 \right\}$.
Show that
\[
R\left( \hat{\theta }_n \right) \xrightarrow[n \to \infty]{\mathbb{P} } R\left( \theta _0 \right).
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item Note that $\mathcal{G} $ is VC-graph, because it is as simple as half-spaces. Note also that we can find an envelope $\sup _{\left\Vert \theta \right\Vert_2 \le 1 } \left| \gamma _\theta \right|$ which is integrable, since
\[\begin{aligned}
\sup _{ \left\Vert \theta \right\Vert _2 \le 1} P\left| \gamma _\theta \right| = \sup _{\left\Vert \theta \right\Vert _2 \le 1}\left\Vert -YX\theta \right\Vert_1 \stackrel{\text{Cauchy-Schwarz} }{\le} \sqrt{\left\Vert X \right\Vert _2 \left\Vert Y\theta \right\Vert _{2}} \stackrel{\left\Vert \theta \right\Vert _2 \le 1}{\le} \sqrt{\left\Vert X \right\Vert _2}
\end{aligned}\]
is finite, because $X$ has a finite second moment. So, $\mathcal{G} $ is Glivenko-Cantelli.
\item Since the class of loss functions is Glivenko-Cantelli, we get that
\[
R\left( \hat{\theta }_n \right) \xrightarrow[n \to \infty]{\mathbb{P} } R\left( \theta _0 \right).
\]
\end{enumerate}
[/latex]

Front: [latex]
Consider the problem of regression, where $\left( X, Y \right)$, $X \in \X$, $Y \in \R$. Take standard regression assumptions, so that
\[
Y = g_0(X) + \xi, \Exnb \xi = 0, \var{\xi} :=\sigma ^{2}<\infty
\] with $\xi$ and $X$ independent. Let $g_0 \in \mathcal{G} $ for some class of functions $\mathcal{G} $, and take the squared error loss
\[
\gamma _g \left( x, y \right) = \left( y - g(x) \right)^{2}.
\]
Show that
\[
R\left( g \right) = \sigma ^{2} + P\left( g - g_0 \right)^{2}.
\]
[/latex]
Back: [latex]
\[\begin{aligned}
R\left( g \right) &= \Exnb \left( Y - g(X) \right)^{2} \\
&= \Exnb \left( \xi - \left( g - g_0 \right)(X) \right)^{2}\\
&= \Exnb \xi^{2} - \underbrace{2 \Exnb \xi \left( g - g_0 \right)(X)}_{=0} + \Exnb \left( g - g_0 \right)^{2}(X)\\
&=\sigma ^{2} + P\left( g - g_0 \right)^{2}.
\end{aligned}\]
[/latex]

Front: [latex]
Consider the problem of regression, where $\left( X, Y \right)$, $X \in \X$, $Y \in \R$. Take standard regression assumptions, so that
\[
Y = g_0(X) + \xi, \Exnb \xi = 0, \var{\xi} :=\sigma ^{2}<\infty
\] with $\xi$ and $X$ independent. Let $g_0 \in \mathcal{G} $ for some class of functions $\mathcal{G} $, and take the squared error loss
\[
\gamma _g \left( x, y \right) = \left( y - g(x) \right)^{2}.
\]
Show that if
\begin{enumerate}[(i)]
\item the $L^{2} \left( \hat{P}_n \right)$ covering numbers do not grow exponentially fast
\[
\frac{1}{n} \log N_2\left( \delta , \G, \hat{P}_n \right) \xrightarrow[n \to \infty]{\P } 0,
\]
\item the envelope is
\[
G := \sup _{g \in \G} \left| g \right| \in L_2(P)
\]
\end{enumerate}
then we have a uniform law of large numbers, so
\[
\sup _{g \in \G} \left| \hat{R}_n(g) - R(g) \right|\xrightarrow[n \to \infty]{\P} 0.
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item
We want to show that if we can find a covering of $\mathcal{G} $ in $L^{2} $ norm, then we can use it to cover $\left\{ \gamma_g : g \in \G \right\} $ in $L^{1} $ norm. For that, we will bound the $L^{1} $ norm distance between $\gamma _g$ and $\gamma _{\tilde{g}}$ by the $L^{2} $ norm distance between $g$ and $\tilde{g}$ multiplied by some constant $M$.
\item To achieve that, we will expand the loss function definition and use Cauchy-Schwarz
\[\begin{aligned}
\hat{P}_n \left| \gamma_g - \gamma_{\tilde{g}} \right| &= \hat{P}_n \left| \left( Y - g \right)^{2} - \left( Y - \tilde{g} \right)^{2} \right| \\
&= \hat{P}_n \left| \left( \xi + \left[ g_0 - g \right] \right)^{2} - \left( \xi + \left[ g_0 - \tilde{g} \right] \right)^{2} \right| \\
&= \hat{P}_n \left| 2 \xi \left( g_0 - g \right) + \left( g_0 - g \right)^{2} - 2\xi \left( g_0 - \tilde{g} \right) - \left( g_0 - \tilde{g}\right)^{2} \right|\\
&= \hat{P}_n \left| -2\xi g + 2\xi \tilde{g} -2g_0g + 2g_0\tilde{g} +g^{2} - \tilde{g}^{2} \right|\\
&\le \hat{P}_n \left| \left( \tilde{g} - g \right)\left( 2 \xi + 2\left| g_0 \right| + \left| g \right| + \left| \tilde{g} \right| \right) \right|\\
&\le \sqrt{\smash[b]{2 \hat{P}_n \underbrace{\left[ \xi^{2} + \left( 2 \left| g_0 \right| + \left| g \right| + \left| \tilde{g} \right| \right)^{2} \right]}_{\le \overline{G}^{2} \text{ for some envelope in $L^{2} $ by (ii)} } }}\sqrt{\hat{P}_n \left( \tilde{g} - g \right)^{2}} &\text{(Cauchy-Schwarz)}
\end{aligned}\]
\vspace{0.3cm}
\item So, we get $M = \sqrt{2\hat{P}_n \overline{G}} $, which will converge to its expectation because the envelope $\overline{G}$ is square integrable. So for any $\delta $, we can say
\[\begin{aligned}
N_1 \left( \sqrt{2 \hat{P}_n \overline{G}^{2}} \delta , \left\{ \gamma_g : g \in \G \right\}, \hat{P}_n \right) &\le N_2 \left( \delta , \G, \hat{P}_n \right). \\
\end{aligned}\]
\item The above lets us show that the entropy limit condition for the family of loss functions to be Glivenko-Cantelli is fulfilled. For any $\delta >0$,
\[\begin{aligned}
\frac{1}{n}H_1 \left( \delta , \left\{ \gamma_g : g \in \G \right\}, \hat{P}_n \right) &= \frac{1}{n}\log N_1 \left( \delta , \left\{ \gamma_g : g \in \G \right\}, \hat{P}_n \right) \\
&\le \frac{1}{n}\log N_2 \left(\delta / \sqrt{2 \hat{P}_n \overline{G}^{2}}, \mathcal{G} , \hat{P}_n \right)\\
&\xrightarrow[n \to \infty]{\P} 0\text{ by assumption (i).}
\end{aligned}\]
\item Now we want to show that the envelope integrability condition is fulfilled, i.e. that the envelope $\Gamma := \sup _{g \in \mathcal{G} }\left| \gamma _g \right|$ satisfies $P\Gamma < \infty$. We know that $G:=\sup _{g \in \mathcal{G} } \left| g \right| \in L^{2} \left( P \right)$, so we can expand $\gamma _g$ to say
\[\begin{aligned}
\sup _{g \in \mathcal{G}} P\left| \gamma _g \right|&= \sup _{g \in \mathcal{G} }\Exnb \left[ \left( Y - g \right)^{2} \right] \\
&= \sup _{g \in \mathcal{G} }\sigma ^{2} + P\left( g - g_0 \right)^{2} &\text{($\Exnb \xi = 0$)} \\
&\le \sup _{g \in \mathcal{G} }\sigma ^{2} + Pg^{2} + Pg_0^{2} \\
&\le \sigma ^{2} + 2PG^{2} \\
&< \infty. &\text{($G \in L^{2}\left( P \right)$ by assumption (ii))}
\end{aligned}\]
\item Since the entropy limit and the envelope integrability conditions hold, we can conclude that the family of loss functions $\left\{ \gamma_g : g \in \G \right\}$ is Glivenko-Cantelli, finishing the proof
\[
\sup _{g \in \mathcal{G} } \left| \hat{R}_n(g) - R(g) \right| = \sup _{g \in \mathcal{G} }\left| \left( \hat{P}_n - P \right)\gamma _g \right| \xrightarrow[n \to \infty]{\mathbb{P} } 0.
\]
\end{enumerate}
[/latex]

Front: [latex]
Consider the class $\mathcal{G} $ of monotone increasing functions from $\mathbb{R} $ to $[0, 1]$ \[\G = \left\{ g : \R \to \left[ 0, 1 \right], g \text{ increasing} \right\}.\] Recall that in this case $H_\infty \left( \delta , \G, \hat{P}_n \right) \le \frac{1}{\delta } \log \left( n + \frac{1}{\delta } \right).$
Show that for least squares monotone function estimator
\[
\hat{g}_n = \arg \min _{g \in \mathcal{G}} \frac{1}{n}\sum_{}^{} \left( Y_i - g\left( X_i \right) \right)^{2}
\] it is true that $\sup _{g \in \mathcal{G} }\left| \hat{R}_n(g) - R(g) \right|\xrightarrow[n \to \infty]{\mathbb{P} } 0.$
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item By the given hint, we can tell that the $L^{2} \left( \hat{P}_n \right)$ covering numbers do not grow exponentially fast, since
\[\begin{aligned}
\frac{1}{n}\log N_2\left( \delta , \mathcal{G} , \hat{P}_n \right) &\le \frac{1}{n}\log N_\infty\left( \delta , \mathcal{G} , \hat{P}_n \right)\\
&= \frac{1}{n}H_\infty \left( \delta , \G, \hat{P}_n \right)\\
&\le \frac{1}{n} \cdot \frac{1}{\delta } \log \left( n + \frac{1}{\delta } \right) \xrightarrow[n \to \infty]{} 0.
\end{aligned}\]
\item Furthermore, the envelope $G := \sup _{g \in \mathcal{G} }\left| g \right| = 1 \in L^{2} \left( P \right)$ is square $P$-integrable.
\item In the least squares loss case, the two above conditions imply that
\begin{enumerate}[(1)]
\item the covering number of the loss functions does not grow exponentially fast
\[
\frac{1}{n} H_1\left( \delta , \left\{ \gamma _g : g \in \mathcal{G} , \hat{P}_n \right\} \right) = \frac{1}{n}\log N_1\left( \delta , \left\{ \gamma _g : g \in \mathcal{G} \right\}, \hat{P}_n \right) \xrightarrow[n \to \infty]{\mathbb{P} } 0,
\]
\item and the envelope of the loss functions $\Gamma := \sup _{g \in \mathcal{G} } \left| \gamma _g \right|$ is $P$-integrable.
\end{enumerate}
\item The above step hence implies
\[
\sup _{g \in \mathcal{G} }\left| \hat{R}_n(g) - R(g) \right|\xrightarrow[n \to \infty]{\mathbb{P} } 0.
\]
\end{enumerate}
[/latex]

Front: [latex]
Consider a parameter space $\Theta $ and an M-estimator $\hat{\theta }_n $ of $\theta _0$. Suppose:
\begin{enumerate}[(1)]
\item $\left( \Theta , \left\Vert \cdot \right\Vert \right)$ is convex, $\left\Vert \cdot \right\Vert $ is some norm.
\item (Identifiability) For all $\delta > 0$, \[\inf_{\left\Vert \theta - \theta _0 \right\Vert > \delta} R\left( \theta \right) - R\left( \theta _0 \right) > 0.\]
\item $\theta \mapsto \gamma_\theta (x)$ is continuous for all $x$, and convex.
\item (Local compactness) There exists an $\varepsilon > 0$ such that the neighbourhood of the true parameter $\Theta _\varepsilon := \left\{ \theta \in \Theta : \left\Vert \theta - \theta _0 \right\Vert \le \varepsilon \right\}$ is compact with envelope
\[
G_\varepsilon := \sup _{\theta \in \Theta _\varepsilon } \left| \gamma_\theta \right| \in L_1\left( P \right).
\]
\end{enumerate}
Show that the above imply that the M-estimator converges to its target \[\left\Vert \hat{\theta }_n -\theta _0 \right\Vert \xrightarrow[n \to \infty]{\P} 0.\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item Let $\alpha := \frac{\varepsilon }{\varepsilon + \left\Vert \hat{\theta }_n - \theta _0 \right\Vert }$ and $\tilde{\theta }_n := \alpha \hat{\theta }_n + \left( 1 - \alpha \right)\theta _0$.
\item Now see that $\tilde{\theta }_n$ is in the locally compact neighbourhood $\Theta _\varepsilon $ because
\[\begin{aligned}
\left\Vert \tilde{\theta}_n - \theta _0 \right\Vert &= \left\Vert \alpha \hat{\theta }_n + \left( \cancel{1} - \alpha \right)\theta _0 - \cancel{\theta _0} \right\Vert \\
&= \alpha \left\Vert \hat{\theta }_n - \theta _0 \right\Vert \\
&= \varepsilon \cdot \frac{\left\Vert \hat{\theta }_n - \theta _0 \right\Vert }{\varepsilon + \left\Vert \hat{\theta }_n - \theta _0 \right\Vert } \le \varepsilon
\end{aligned}\]
\item Note that the compact neighbourhood of the target $\Theta _\varepsilon $ satisfies conditions for entropy with bracketing of the losses to be finite:
\begin{enumerate}[(a)]
\item $\Theta _\varepsilon $ is compact,
\item $\theta \mapsto \gamma _\theta (x)$ is continuous for all $x$,
\item $\Theta _\varepsilon $ has an envelope in $L_1\left( P \right)$.
\end{enumerate}
\item
Since the entropy with bracketing is finite for the loss functions, we have in the neighbourhood of the target that
\[
\sup _{\theta \in \Theta _\varepsilon } \left| \hat{R}_n(\theta ) - R(\theta ) \right| \xrightarrow[n \to \infty]{\mathbb{P} } 0.
\]
\item Now we can use the above fact of convergence in $\Theta _\varepsilon $ together with convexity and our construction of $\tilde{\theta }_n$ to show that it converges
\[\begin{aligned}
0 &\le R\left( \tilde{\theta}_n \right) - R\left( \theta _0 \right) \\
&= \overbrace{\underbrace{\left[ \hat{R}_n(\theta _0) - R(\theta _0) \right]}_{\xrightarrow[n \to \infty]{\mathbb{P} } 0} - \underbrace{\left[ \hat{R}_n(\tilde{\theta}_n) - R(\tilde{\theta}_n) \right]}_{\xrightarrow[n \to \infty]{\mathbb{P} } 0}}^{:=K_n} + \hat{R}_n \left( \tilde{\theta}_n \right) - \hat{R}_n \left( \theta _0 \right) \\
&\le K_n + \alpha \hat{R}_n \left( \hat{\theta }_n \right) + \left( 1 - \alpha \right) \hat{R}_n \left( \theta _0 \right) - \hat{R}_n (\theta _0) &\text{(Convex loss)} \\
&= K_n + \alpha \left\{ \smash[b]{\underbrace{\hat{R}_n \left( \hat{\theta }_n \right) - \hat{R}_n \left( \theta _0 \right)}_{\le 0 \text{ since $\hat{\theta }_n $ is minimizer} }} \right\} \xrightarrow[n \to \infty]{\mathbb{P} } 0.
\end{aligned}\]
\vspace{0.3cm}
\item Since the above step proves $R\left( \tilde{\theta}_n \right) \xrightarrow[n \to \infty]{\P} R(\theta _0)$, by identifiability assumption we get convergence in the norm
\[
\left\Vert \tilde{\theta _n} - \theta _0 \right\Vert \xrightarrow[n \to \infty]{\P} 0.
\]
\item We can now use the convergence of $\tilde{\theta }_n$ in the neighbourhood of $\theta _0$ to prove convergence of $\hat{\theta }_n $ globally. We just need to rearrange terms in the equation from step 2. Observe that
\[\begin{aligned}
\left\Vert \tilde{\theta}_n - \theta _0 \right\Vert &= \frac{\varepsilon \left\Vert \hat{\theta }_n - \theta _0 \right\Vert }{\varepsilon + \left\Vert \hat{\theta }_n - \theta _0 \right\Vert }\\
\implies \left\Vert \hat{\theta }_n - \theta _0 \right\Vert &= \frac{\varepsilon \left\Vert \tilde{\theta}_n - \theta _0 \right\Vert}{\varepsilon - \left\Vert \tilde{\theta}_n - \theta _0 \right\Vert } \xrightarrow[n \to \infty]{\P} 0.
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Consider the problem of finding the median. $\X = \R$, $\Theta =\R$, $\gamma_\theta \left( x \right) = \left| x - \theta \right|$. Assume $\Exnb \left| X \right| < \infty$.
How can you show that the sample median converges to the true one $\hat{\theta }_n \xrightarrow[n \to \infty]{\P} \theta _0$?
[/latex]
Back: [latex]
We know that there is no integrable envelope for the class of losses $\left\{ \gamma _\theta : \theta \in \Theta \right\}$, so we can't use the nice proposition relying on the family being Glivenko-Cantelli. However, the median satisfies the conditions for convergence with local compactness:
\begin{enumerate}[(1)]
\item $\Theta $ is convex.
\item (Identifiability) For all $\delta > 0$, \[\inf_{\left\Vert \theta - \theta _0 \right\Vert > \delta} R\left( \theta \right) - R\left( \theta _0 \right) > 0.\]
\item $\theta \mapsto \gamma_\theta (x) = \left| x - \theta \right|$ is continuous for all $x$, and convex.
\item (Local compactness) There exists an $\varepsilon > 0$ such that the neighbourhood of the true parameter $\Theta _\varepsilon := \left\{ \theta \in \Theta : \left\Vert \theta - \theta _0 \right\Vert \le \varepsilon \right\}$ is compact with envelope
\[
G_\varepsilon := \sup _{\theta \in \Theta _\varepsilon } \left| \gamma_\theta \right| \in L_1\left( P \right).
\]
\end{enumerate}
So, we can conclude
\[
\hat{\theta }_n \xrightarrow[n \to \infty]{\P} \theta _0.
\]
[/latex]

Front: [latex]
Let $\left( Z_{n} \right)_{n \geq 1} $, $Z$ be random variables in $\mathbb{R} ^{r}$. What does it mean for $\left( Z_{n} \right)_{n \geq 1} $ to converge to $Z$ in distribution?
[/latex]
Back: [latex]
$\left( Z_{n} \right)_{n \geq 1} $ converge in distribution to $Z$ if $f : \mathbb{R} ^{r} \to \mathbb{R} $ we have
\[
\lim_{n \to \infty} \Exnb f\left( Z_n \right) \to \Exnb f\left( Z \right)
\]for all bounded and continuous functions $f : \mathbb{R}^{r} \to \mathbb{R} $.
[/latex]

Front: [latex]
Let $\left( Z_{n} \right)_{n \geq 1} $, $Z$ be random variables. What does it mean if \[Z_n \xrightarrow[n \to \infty]{\mathcal{L} } Z?\]
[/latex]
Back: [latex]
$ Z_n \xrightarrow[n \to \infty]{\mathcal{L} } Z$ means that $Z_n$ converge in distribution to $Z$, meaning that
\[
\lim_{n \to \infty} \Exnb f\left( Z_n \right) \to \Exnb f\left( Z \right)
\]for all bounded and continuous functions $f$.
[/latex]

Front: [latex]
Let $\left( Z_{n} \right)_{n \geq 1} $ and $Z$ be random variables, and let $F_Z$ be the distribution function of $Z$. What kind of convergence allows us to say
\[
\P\left( Z_n \le z \right) \xrightarrow[n \to \infty]{} \P\left( Z \le z \right)
\]for all $F_Z$-continuity points?
[/latex]
Back: [latex]
By the Portmanteau Theorem, this is an equivalent condition to convergence in distribution
\[
Z_n \xrightarrow[n \to \infty]{\L} Z.
\]
Actually, this is also true for the other types of convergence (almost sure, in probability, in $L^{p}$), since they imply convergence in distribution.
[/latex]

Front: [latex]
Suppose $X_1, \dots, X_n, \dots$ are i.i.d. copies of $X \in \R$. Consider the average \[\overline{X}_n = \frac{1}{n}\sum_{i = 1}^{n} X_i.\] What are $\Exnb \overline{X}_n$ and $\varnb \left( \overline{X}_n \right)$?
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item $\Exnb \overline{X}_n = \mu$,
\item $\varnb \left( \overline{X}_n \right) = \frac{\sigma ^{2}}{n}$.
\end{enumerate}
[/latex]

Front: [latex]
State the Central Limit Theorem in $\mathbb{R} $.
[/latex]
Back: [latex]
Suppose $X_1, \dots, X_n, \dots$ are i.i.d. copies of $X \in \R$. Let $\Exnb X = \mu$ and $\var{X} = \sigma ^{2} < \infty$. Then for the mean $\overline{X}_n$ we have
\[
\frac{\overline{X}_n - \mu }{\sigma /\sqrt{n} } = \sqrt{n} \frac{\left( \overline{X}_n - \mu \right)}{\sigma } \xrightarrow[n \to \infty]{\L} \No\left( 0, 1 \right).
\] or, in other words,
\[
\P \left( \frac{\sqrt{n} \left( \overline{X}_n - \mu \right)}{\sigma } \le z \right) \xrightarrow[n \to \infty]{} \Phi \left( z \right) \text{ for all $z \in \R$} .
\]
[/latex]

Front: [latex]
Let $X_1, \dots, X_n $ be i.i.d. $\mathbb{R} ^{r}$-valued random variable copies of $X$, with expectation $\mu = \Exnb X$. How is the covariance matrix of $X$ defined?
[/latex]
Back: [latex]
The covariance matrix $\Sigma $ is
\[
\Sigma := \Exnb XX^{T} - \mu \mu ^{T}.
\]
[/latex]

Front: [latex]
State the Central Limit Theorem in $\mathbb{R} ^{r}$.
[/latex]
Back: [latex]
Let $X_1, \dots, X_n $ be i.i.d. $\mathbb{R} ^{r}$-valued random variable copies of $X$, with expectation $\mu = \Exnb X$ and covariance matrix $\Sigma = \Exnb XX^{T}- \mu \mu ^{T}$. Then we have
\[
\sqrt{n} \left( \overline{X}_n - \mu \right) \xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, \Sigma \right).
\]
[/latex]

Front: [latex]
Let $X_1, \dots, X_n $ be i.i.d. $\mathbb{R} ^{r}$-valued random variable copies of $X$, with expectation $\mu = \Exnb X$ and covariance matrix $\Sigma = \Exnb XX^{T}- \mu \mu ^{T}$. Take $a \in \mathbb{R} ^{r}$. What does
\[
\sqrt{n} \left[ a^{T}\left( \overline{X}_n - \mu \right) \right]
\] converge to?
[/latex]
Back: [latex]
By the Central Limit Theorem in $\mathbb{R} ^{r}$,
\[
\sqrt{n} \left[ a^{T}\left( \overline{X}_n - \mu \right) \right] \xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, a^{T}\Sigma a \right)
\]
[/latex]

Front: [latex]
Consider the empirical and true distribution functions $\hat{F}_n $ and $F$. Define \[W_n(t) := \sqrt{n} \left( \hat{F}_n(t) - F(t) \right).\]
What is the asymptotic distribution of $W_n(t)$ as $n \to \infty$ for some fixed value of $t$?
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item We can find the asymptotic distribution of $W_n(t)$ by using the Central Limit Theorem. To find the asymptotic variance, we need to consider $\hat{F}_n $ as a mean of random variables whose variance we can compute. As it is, $\hat{F}_n $ is a normalized sum of identity functions, which are $\text{Bernoulli} \left( F(t) \right)$ distributed with variance $F(t)\left( 1 - F(t) \right)$.
\item So, we can conclude
\[
W_n\left( t \right) \xrightarrow[n \to \infty]{\L} \No\left( 0, F(t)\left( 1 - F(t) \right) \right) \text{ for all }t.
\]
\end{enumerate}
[/latex]

Front: [latex]
Let $\mathcal{X} = \mathbb{R} $, and consider the problem of estimating the distribution function $F$ given $X_1, \dots, X_n $. Define for all $t \in \mathbb{R} $ the stochastic process \[W_n \left( t \right) := \sqrt{n} \left( \hat{F}_n(t) - F(t) \right).\] Now take $s < t$. What does the random vector $\begin{pmatrix} W_n \left( s \right)\\ W_n \left( t \right)\end{pmatrix}$ converge to?
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item We know by the Central Limit Theorem that
\[
\begin{pmatrix} W_n \left( s \right)\\ W_n \left( t \right)\end{pmatrix} \xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, \Sigma \left( s, t \right) \right)
\] for some covariance matrix $\Sigma \left( s, t \right)$.
\item To find the covariance matrix, we need to consider $\hat{F}_n $ as a mean of random variables whose covariance we can compute
\[
\hat{F}_n (t) := \frac{1}{n} \sum_{i = 1}^{n} 1_{\left\{ X_i \le t \right\}}.
\] So, for a fixed $s, t$ with $s < t$, take $T = 1_{\left\{ X \le t \right\}}$, $S = 1_{\left\{ X \le s \right\}}$.
\item The asymptotic covariance $\Sigma $ is
\[
\Sigma \left( s, t \right) = \covnb \begin{pmatrix}
S \\
T
\end{pmatrix} = \begin{bmatrix}
\varnb \left( S \right) & \covnb \left( S, T \right) \\
\covnb \left( S, T \right) &\varnb \left( T \right)
\end{bmatrix} = \begin{bmatrix}
F(s)\left( 1 - F(s) \right) & F(s)\left( 1 - F(t) \right)\\
F(s)\left( 1 - F(t) \right) & F(t)\left( 1 - F(t) \right)\\
\end{bmatrix}.
\]
\item So, by CLT
\[
\begin{pmatrix} W_n \left( s \right)\\ W_n \left( t \right)\end{pmatrix} = \sqrt{n} \begin{pmatrix}
\hat{F}_n(s) - F(s) \\
\hat{F}_n(t) - F(t)
\end{pmatrix} \xrightarrow[n \to \infty]{\L} \No \left( 0, \begin{pmatrix}
F(s)\left( 1 - F(s) \right) & F(s)\left( 1 - F(t) \right) \\
F(s)\left( 1 - F(t) \right) & F(t)\left( 1 - F(t) \right)
\end{pmatrix} \right).
\]
\end{enumerate}
[/latex]

Front: [latex]
Let $\mathcal{K} _0$ be the family of all bounded functions $K : \left[ 0, 1 \right]\to \mathbb{R} $. What is a (standard) Brownian bridge $B \in \mathcal{K} _0$?
[/latex]
Back: [latex]
The (standard) Brownian bridge $B(\cdot) \in \mathcal{K} _0$ is a stochastic process such that
\begin{enumerate}[(1)]
\item $B\left( 0 \right) = B\left( 1 \right) = 0$,
\item for all $r \ge 1$ and all $t_1, \dots, t_r \in \left( 0, 1 \right)$, the vector $\begin{pmatrix}
B\left( t_1 \right) \\
\vdots \\
B\left( t_r \right)
\end{pmatrix}$ is multivariate normal with mean zero,
\item for all $s \le t$, $\cov{B(s), B(t)} = s\left( 1 - t \right)$,
\item the sample paths of $B$ are almost surely continuous.
\end{enumerate}
[/latex]

Front: [latex]
Consider the empirical and true distribution functions $\hat{F}_n $ and $F$. Define \[W_n(t) := \sqrt{n} \left( \hat{F}_n(t) - F(t) \right),\] and take $s \le t$. By the Central Limit Theorem, we know that
\[
\begin{pmatrix} W_n \left( s \right)\\ W_n \left( t \right)\end{pmatrix} \xrightarrow[n \to \infty]{\L} \No \left( 0, \begin{pmatrix}
F(s)\left( 1 - F(s) \right) & F(s)\left( 1 - F(t) \right) \\
F(s)\left( 1 - F(t) \right) & F(t)\left( 1 - F(t) \right)
\end{pmatrix} \right).
\]
Let $B$ be a Brownian bridge. What distribution must the distribution function $F$ refer to for us to be able to say
\[
\begin{pmatrix} W_n \left( s \right)\\ W_n \left( t \right)\end{pmatrix} \xrightarrow[n \to \infty]{\mathcal{L} } \begin{pmatrix}
B\left( s \right) \\
B\left( t \right)
\end{pmatrix}?
\]
[/latex]
Back: [latex]
$F$ must be the distribution function of the uniform distribution on $[0, 1]$.
\begin{enumerate}[1.]
\item We know that for $0< s \le t < 1$ the vector $\begin{pmatrix}
B\left( s \right) \\
B\left( t \right)
\end{pmatrix}$ is multivariate normal with mean zero and covariance $\begin{pmatrix}
s\left( 1 - s \right) & s\left( 1 - t \right)\\
s\left( 1 - t \right) &t\left( 1 - t \right)
\end{pmatrix}.$
\item So, for the covariance above to be the limiting covariance of $ \begin{pmatrix} W_n \left( s \right)\\ W_n \left( t \right)\end{pmatrix}$, $F$ must be an identity function from $[0,1]$ to $[0,1]$, so it is the distribution function of $\text{Uniform} \left( 0, 1 \right)$.
\end{enumerate}
[/latex]

Front: [latex]
State Donsker's theorem.
[/latex]
Back: [latex]
Take an empirical distribution function $\hat{F}_n $ and its corresponding true function $F$. Define for all $t$, \[W_n(t) := \sqrt{n} \left( \hat{F}_n(t) - F(t) \right).\] Let $B$ be the standard Brownian bridge, and let for all $t$, \[W_F\left( t \right) = B\left( F\left( t \right) \right).\] Then it is true that
\[
W_n \xrightarrow[n \to \infty]{\mathcal{L} } W_F.
\]
[/latex]

Front: [latex]
Why is Donsker's theorem, given below, relevant for statistics?
\paragraph{Donsker's theorem.} Take an empirical distribution function $\hat{F}_n $ and its corresponding true function $F$. Define for all $t$, \[W_n(t) := \sqrt{n} \left( \hat{F}_n(t) - F(t) \right).\] Let $B$ be the standard Brownian bridge, and let for all $t$, $W_F\left( t \right) = B\left( F\left( t \right) \right).$ Then it is true that
$W_n \xrightarrow[n \to \infty]{\mathcal{L} } W_F.$
[/latex]
Back: [latex]
Donsker's theorem allows us to analyse the limiting distribution of a stochastic process $W_n$ by a distribution $W_F$ we know. This can be used to, for example, test the hypothesis $H_0 : F = F_0$.
[/latex]

Front: [latex]
Consider a continuous distribution $\mathcal{D} $ with distribution function $F$. Define for all $t$, \[W_n(t) := \sqrt{n} \left( \hat{F}_n(t) - F(t) \right).\] What can you say about the continuity of $W_n$ and its distribution limit $W_F$? What does this mean for the sequence $\left( W_{n} \right)_{n \geq 1} $?
[/latex]
Back: [latex]
$W_n$ is not continuous, but the limit $W_F$ is. The sequence $\left( W_{n} \right)_{n \geq 1} $ is asymptotically continuous.
[/latex]

Front: [latex]
Consider a function $g \in \mathcal{G} $, and let $P$ be some probability measure. What is the empirical process $\nu_n(g)$ indexed by $g$?
[/latex]
Back: [latex]
The empirical process $\nu_n : \G \to \R$ is
\[
\nu_n(g) = \sqrt{n} \left( \hat{P}_n - P \right)g.
\]
[/latex]

Front: [latex]
What is a $P$-Brownian bridge?
[/latex]
Back: [latex]
Let $\mathcal{G} $ be a class of functions $g : \mathcal{X} \to \mathbb{R} $ equipped with a metric induced by the norm $L^{2} \left( P \right)$ for some probability measure $P$, and let $\mathcal{K} $ be the space of all bounded functions $K : \mathcal{G} \to \mathbb{R} $. A $P$-Brownian bridge $\nu$ is a random element $\nu \in \K$ with
\begin{enumerate}[(1)]
\item for all $r \in \N$, $\begin{pmatrix}
g_1 \\
\vdots \\
g_r
\end{pmatrix}$, $\begin{pmatrix}
\nu(g_1) \\
\vdots \\
\nu(g_r)
\end{pmatrix}$ is multivariate normal,
\item $\cov{\nu\left(g \right), \nu\left( \tilde{g} \right)} = Pg\tilde{g} - PgP\tilde{g}$, and
\item $\nu$ is almost surely continuous.
\end{enumerate}
[/latex]

Front: [latex]
Let $\mathcal{G} $ be a class of functions $g : \mathcal{X} \to \mathbb{R} $, and let $\mathcal{K} $ be the space of all bounded functions $K : \mathcal{G} \to \mathbb{R} $.
For an element $K \in \mathcal{K} $, what does $\left\Vert K \right\Vert _{\mathcal{G} }$ mean in this case?
[/latex]
Back: [latex]
\[
\left\Vert K \right\Vert _{\mathcal{G} } = \sup _{g \in \mathcal{G} }\left| K\left( g \right) \right|
\]
[/latex]

Front: [latex]
Let $\mathcal{G} $ be a class of functions $g : \mathcal{X} \to \mathbb{R} $ equipped with a metric induced by the norm $L^{2} \left( P \right)$ for some probability measure $P$, and let $\mathcal{K} $ be the space of all bounded functions $K : \mathcal{G} \to \mathbb{R} $ equipped with a metric induced by the norm $\left\Vert K \right\Vert _\mathcal{G} $. Take $\nu \in \mathcal{K} $.
What does it mean for a sequence $\nu(g_n)$ to converge to $\nu(g)$ in this case? Expand the whole $\varepsilon $-$\delta $ definition of limits along with how you could compute the necessary distances, and explain why it's relevant for the definition of $P$-Brownian bridges.
[/latex]
Back: [latex]
$\nu(g_n)$ converges to $\nu(g)$ if for all $\varepsilon > 0$ there exists a $\delta > 0$ such that if
\[
\left\Vert g_n - g \right\Vert ^{2} = P\left( g_n - g \right)^{2} < \delta \text{ then } \left| \nu\left( g_n \right) - \nu\left( g \right) \right| < \varepsilon .
\]
This is relevant for the definition of $P$-Brownian bridges, because a $P$-Brownian bridge $\nu$ is almost surely continuous in the above sense. So, with probability 1, for any sequence $\left( g_{n} \right)_{n \geq 1} $ with $\lim_{n \to \infty} P\left( g_n - g \right)^{2} = 0$ we can say $\lim_{n \to \infty} \nu\left( g_n \right) = \nu\left( g \right)$.
[/latex]

Front: [latex]
Let $\mathcal{G} $ be a class of functions $g : \mathcal{X} \to \mathbb{R} $, and let $\mathcal{K} $ be the space of all bounded functions $K : \mathcal{G} \to \mathbb{R} $, and finally let $P$ be a distribution over $\mathcal{K} $.
What does it mean for $\mathcal{G} $ to be a $P$-Donsker class?
[/latex]
Back: [latex]
We call $\mathcal{G} $ a $P$-Donsker class if for any $g \in \mathcal{G} $,
\[\nu_n(g) \xrightarrow[n \to \infty]{\mathcal{L} } \nu(g)\]
where $\nu_n$ is an empirical process and $\nu$ is the $P$-Brownian bridge, both indexed by $g$.
[/latex]

Front: [latex]
What does it mean for a process $\nu_n : \mathcal{G} \to \mathbb{R} $ to be asymptotically continuous at $g_0 \in \mathcal{G} $? Explain which norms are typically used in this definition.
[/latex]
Back: [latex]
$\nu_n$ is asymptotically continuous at $g_0$ if for all sequences $\left( g_{n} \right)_{n \geq 1} $ with
\[
\left\Vert g_n - g_0 \right\Vert \xrightarrow[n \to \infty]{\mathbb{P} } 0
\]it's true that
\[
\left|\nu_n \left( g_n \right) - \nu_n(g_0) \right| \xrightarrow[n \to \infty]{\P} 0
\] converges as well.
Usually, the norm we use on $\mathcal{G} $ is $L^{2} \left( P \right)$ for some probability measure $P$.
[/latex]

Front: [latex]
Prove the following implication, where $\left\Vert g \right\Vert := \sqrt{Pg^{2}} $, $\left\Vert g \right\Vert _n :=\sqrt{\hat{P}_n g^{2}} $:
\[
\begin{cases}
\frac{1}{n} \log N_2 \left( \delta , \G, \hat{P}_n \right)\xrightarrow[n \to \infty]{\P} 0 \text{ for all $\delta >0$}\\
G \in L_2\left( P \right)
\end{cases} \implies \sup _{g \in \G} \left| \left\Vert g \right\Vert _n ^{2}- \left\Vert g \right\Vert ^{2} \right| \xrightarrow[n \to \infty]{\P} 0.
\]
[/latex]
Back: [latex]
We want to show that
\[
\sup _{g \in \mathcal{G} } \left| \left\Vert g \right\Vert ^{2}_n - \left\Vert g \right\Vert ^{2} \right| = \sup _{g \in \mathcal{G} } \left| \hat{P}_n g^{2} - Pg^{2} \right| \xrightarrow[n \to \infty]{\mathbb{P} } 0.
\] For that purpose we will show that the family $\mathcal{G} ^{2} := \left\{ g^{2} : g \in \mathcal{G} \right\}$ of squared $g$s satisfies the $\frac{1}{n}$ entropy limit and envelope in $L^{1} $ conditions for uniform convergence.
\begin{enumerate}[1.]
\item First, since $G \in L^{2} \left( P \right)$, we can use Cauchy-Schwarz together with the envelope to write for all $g, \tilde{g} \in \mathcal{G} $,
\[\begin{aligned}
\hat{P}_n \left| g^{2} - \tilde{g}^{2} \right| &= \hat{P}_n \left| g - \tilde{g} \right| \overbrace{\left| g + \tilde{g} \right|}^{\le 2G} \\
&\le \underbrace{\sqrt{\hat{P}_n \left| g - \tilde{g} \right|^{2}}}_{=\delta } \sqrt{4\hat{P}_n G^{2}}. &\text{(Cauchy-Schwarz)} \\
\end{aligned}\]
\item Now we can use one of our assumptions to conclude that the covering number of $\left\{ g^{2} : g \in \mathcal{G} \right\}$ does not grow exponentially fast, satisfying the entropy limit condition
\[\begin{aligned}
\frac{1}{n}\log N_1\left( \delta \sqrt{4 \hat{P}_n G^{2}} , \left\{ g^{2} : g \in \mathcal{G} \right\} , \hat{P}_n \right) &\le \frac{1}{n}\log N_2\left( \delta , \G, \hat{P}_n \right) \xrightarrow[n \to \infty]{\mathbb{P} } 0. \\
\end{aligned}\]
\item Now, since $G \in L^{2} \left( P \right)$, we know also that $G^{2} \in L^{1} \left( P \right) $.
\item We know that the two above steps together imply our desired statement
\[
\sup _{g \in \mathcal{G} } \left| \left\Vert g \right\Vert _n^{2} - \left\Vert g \right\Vert ^{2} \right| = \sup _{g \in \mathcal{G} } \left| \left( \hat{P}_n - P \right)g^{2} \right| \xrightarrow[n \to \infty]{\mathbb{P} } 0.
\]
\end{enumerate}
[/latex]

Front: [latex]
What is the symmetrized empirical process $\nu_n^{\sigma }$?
[/latex]
Back: [latex]
The symmetrized empirical process is the empirical process of a symmetrized empirical measure
\[\nu_n^{\sigma}\left( g \right) := \sqrt{n} \hat{P}_n ^{\sigma}g.\]
To be more precise, take a sequence $\left( \sigma _{i} \right)_{i \geq 1} $ of independent Rademacher random variables. Now $\hat{P}_n ^{\sigma}g := \frac{1}{n} \sum_{i = 1}^{n} \sigma _i g\left( X_i \right)$ is the symmetrized empirical measure, and
\[
\nu_n^{\sigma}\left( g \right) = \sqrt{n} \hat{P}_n ^{\sigma}g
\] is the symmetrized empirical process.
[/latex]

Front: [latex]
Prove the implication
\[
\sup _{g \in \G} \left| \left\Vert g \right\Vert _n ^{2}- \left\Vert g \right\Vert ^{2} \right| \xrightarrow[n \to \infty]{\P} 0 \implies
\sup _{g \in \G} \left| \left\Vert g \right\Vert _n - \left\Vert g \right\Vert \right| \xrightarrow[n \to \infty]{\P} 0
\] where $\left\Vert g \right\Vert := \sqrt{Pg^{2}} , \left\Vert g \right\Vert _n := \sqrt{\hat{P}_n g^{2}} $.
[/latex]
Back: [latex]
This is true simply because of continuity of the square root function. For any random sequence $\left( Z_{n} \right)_{n \geq 1} $ if $Z_n \xrightarrow[n \to \infty]{\mathbb{P} } Z$, then by the continuous mapping theorem for any continuous function $f$ we have $f\left( Z_n \right) \xrightarrow[n \to \infty]{\mathbb{P} } f(Z)$. Since $\sqrt{\cdot} $ is continuous, the desired implication follows.
[/latex]

Front: [latex]
Show that if
\begin{enumerate}[(1)]
\item $G \in L_2(P)$,
\item $\frac{1}{n} \log N_2 \left( \delta , \G, \hat{P}_n \right) \xrightarrow[n \to \infty]{\P} 0$, and
\item $\nu_n^{\sigma }\left( g - g_0 \right) \xrightarrow[]{\P} 0$ as $\left\Vert g - g_0 \right\Vert_n \xrightarrow[]{\mathbb{P} } 0$,
\end{enumerate}
then $\nu_n$ is asymptotically continuous, so \[\left|\nu_n \left( g_n - g \right) \right| \xrightarrow[n \to \infty]{\P} 0 \text{ as } \left\Vert g_n - g \right\Vert \xrightarrow[n \to \infty]{\mathbb{P} } 0.\]
Recall that the following implication holds
\[ \begin{cases}
\text{(i)} & \frac{1}{n} \log N_2 \left( \delta , \G, \hat{P}_n \right)\xrightarrow[n \to \infty]{\P} 0 \text{ for all $\delta >0$}\\
\text{(ii)}& G \in L_2\left( P \right)
\end{cases} \implies \sup _{g \in \G} \left| \left\Vert g \right\Vert _n ^{2}- \left\Vert g \right\Vert ^{2} \right| \xrightarrow[n \to \infty]{\P} 0.\]
[/latex]
Back: [latex]
To go from the convergence in condition (3) to asymptotic continuity of $\nu_n$, we need to bound $\nu_n$ by $\nu_n^{\sigma}$ with symmetrization, and to replace the empirical norm of $\left\Vert \cdot \right\Vert _n$ by the theoretical norm $\left\Vert \cdot \right\Vert$.
\begin{enumerate}[]
\item
Using condition (3), take any $\varepsilon > 0$ and choose $\delta > 0$ such that
\[\begin{aligned}
\P \left( \sup _{g \in \G, \left\Vert g - g_0 \right\Vert _n \le \delta } \left| \nu_n^{\sigma }\left( g - g_0 \right) \right| > \frac{\varepsilon }{4} \right) &\xrightarrow[n \to \infty]{} 0.\\
\end{aligned}\]
\item We will begin by the standard symmetrization probability inequality. For brevity, write $\tilde{g} = g - g_0$.
\[\begin{aligned}
\P \left( \sup _{\left\Vert g - g_0 \right\Vert \le \frac{\delta }{2} } \left| \nu_n\left( g - g_0 \right) \right| > \varepsilon \right) &\le 4 \P \left( \sup _{\left\Vert \tilde{g} \right\Vert \le \frac{\delta }{2} } \left| \nu_n^{\sigma }\left( \tilde{g} \right) \right| > \frac{\varepsilon}{4} \right)\\
&\le 4 \P \left( \sup _{\left\Vert \tilde{g} \right\Vert \le \frac{\delta }{2} } \left| \nu_n^{\sigma }\left( \tilde{g} \right) \right| > \frac{\varepsilon}{4}, \sup _{g \in \G} \left| \left\Vert \tilde{g} \right\Vert _n - \left\Vert \tilde{g} \right\Vert \right| \le \frac{\delta }{2} \right) \\
&+ 4 \P \left(\sup _{\left\Vert \tilde{g} \right\Vert \le \frac{\delta }{2} } \left| \nu_n^{\sigma }\left( \tilde{g} \right) \right| > \frac{\varepsilon}{4}, \smash{ \underbrace{\sup _{g \in \G} \left| \left\Vert \tilde{g} \right\Vert _n - \left\Vert \tilde{g} \right\Vert \right| }_{\xrightarrow[n \to \infty]{\mathbb{P} } 0 \text{ by hint} }> \frac{\delta }{2}} \right)
\end{aligned}\]
\vspace{0.3cm}
\item To deal with the remaining term, note that
\[
\sup _{g \in \mathcal{G} } \left| \left\Vert \tilde{g} \right\Vert _n - \left\Vert \tilde{g} \right\Vert \right| \le \frac{\delta }{2} \text{ and } \left\Vert \tilde{g} \right\Vert \le \frac{\delta }{2} \implies \left\Vert \tilde{g} \right\Vert_n \le \delta
\]
which, continuing the inequalities above for large $n$, allows us to conclude
\[\begin{aligned}
\P \left( \sup _{\left\Vert g - g_0 \right\Vert \le \frac{\delta }{2} } \left| \nu_n\left( g - g_0 \right) \right| > \varepsilon \right) &\le 4 \P \left( \sup _{\left\Vert \tilde{g} \right\Vert \le \frac{\delta }{2} } \left| \nu_n^{\sigma }\left( \tilde{g} \right) \right| > \frac{\varepsilon}{4}, \sup _{g \in \G} \left| \left\Vert \tilde{g} \right\Vert _n - \left\Vert \tilde{g} \right\Vert \right| \le \frac{\delta }{2} \right)\\
&\le 4\P \left( \sup _{g \in \G, \left\Vert g - g_0 \right\Vert _n \le \delta } \left| \nu_n^{\sigma }\left( g - g_0 \right) \right| > \frac{\varepsilon }{4} \right)\\
&\xrightarrow[n \to \infty]{} 0. &\text{(Step 1.)}
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $ with radius $R = \sup _{g \in \mathcal{G} } \left\Vert g \right\Vert$ and empirical radius $\hat{R} := \sup _{g \in \G} \left\Vert g \right\Vert _n $.
Take a function $g \in \mathcal{G}$. Describe how $g$ is represented in the chaining procedure by a sum of some functions $g^{S}$.
[/latex]
Back: [latex]
$\G_s := \left\{ g_j^{s} \right\}_{j = 1}^{N_s}$ is the minimal $2^{-s}\hat{R}_n$-covering of $\left( \G, \left\Vert \cdot \right\Vert _n \right)$. $N_s = N_2 \left( 2^{-s}\hat{R}_n, \G, \hat{P}_n \right)$, $H_s := \log N_s$. Note that $N_0 = 1$, $\mathcal{G} _0 = \left\{ 0 \right\}$, $H_0 = 0$.
Fix some large value $S \in \mathbb{N} $. Now define the "chain" of $g^{S + 1}$ by
\begin{center}\begin{tabular}{ c }
$\left\Vert g^{S + 1}-g \right\Vert _n = \min _{g_j \in \G_S} \left\Vert g_j - g \right\Vert _n$\\
\\
$\left\Vert g^{S} - g^{S + 1} \right\Vert _n = \min _{g_j \in \G_{S - 1}}\left\Vert g_j - g^{S + 1} \right\Vert _n$\\
\\
\vdots\\
\\
$g^{0} = \left\{ 0 \right\}$
\end{tabular}\end{center}
Then we can describe $g$ by telescoping as
\[\begin{aligned}
g &= g - g^{S + 1} + g^{S + 1} \\
&= g - g^{S + 1} + g^{S + 1} - g^{S} + g^{S} \\
&= g - g^{S + 1} + \sum_{s = 0}^{S} \left( g^{s + 1} - g^{s} \right).
\end{aligned}\]
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $ with radius $R = \sup _{g \in \mathcal{G} } \left\Vert g \right\Vert$ and empirical radius $\hat{R} := \sup _{g \in \G} \left\Vert g \right\Vert _n $. Take a function $g \in \mathcal{G} $. Chaining allows us to represent it by telescoping as
\[\begin{aligned}
g &= g - g^{S + 1} + g^{S + 1} \\
&= g - g^{S + 1} + g^{S + 1} - g^{S} + g^{S} \\
&= g - g^{S + 1} + \sum_{s = 0}^{S} \left( g^{s + 1} - g^{s} \right).
\end{aligned}\]
Give an upper bound on $\left\Vert g - g^{S + 1} \right\Vert_n $ as well as $\left\Vert g^{s + 1}- g^{s} \right\Vert _n$ for $s \in \left\{ 0, \dots, S \right\}$.
[/latex]
Back: [latex]
The bounds are:
\begin{enumerate}[(1)]
\item For the start of the chain, $\left\Vert g - g^{S + 1} \right\Vert _n \le 2^{-\left( S + 1 \right)}\hat{R}_n $.
\item For each subsequent element with $s \in \left\{ 0, \dots, S \right\},$ $ \left\Vert g^{s + 1} - g^{s} \right\Vert _n \le 2^{-s}\hat{R}_n .$
\end{enumerate}
This comes from the definition of the chain. Each $g^{S + 1}$ is defined by its empirical norm with respect to the others as
\begin{center}\begin{tabular}{ c }
$\left\Vert g^{S + 1}-g \right\Vert _n = \min _{g_j \in \G_S} \left\Vert g_j - g \right\Vert _n$\\
\\
$\left\Vert g^{S} - g^{S + 1} \right\Vert _n = \min _{g_j \in \G_{S - 1}}\left\Vert g_j - g^{S + 1} \right\Vert _n$\\
\\
\vdots\\
\\
$g^{0} = \left\{ 0 \right\}$
\end{tabular}\end{center}
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $ which is uniformly bounded by $K$, i.e. $\sup _{g \in \mathcal{G} } g(x) \le K$ for all $x$. Take a function $g \in \mathcal{G} $. In this case, Hoeffding's inequality for a symmetrized empirical distribution $\hat{P}_n ^{\sigma}$ is
\[
\mathbb{P} \left( \left| \hat{P}_n ^{\sigma}g \right|\ge K \sqrt{\frac{2t}{n}} \right) \le 2\exp \left[ -t \right].
\]
Provide a Hoeffding's inequality bound for the symmetrized empirical process $\nu_n^{\sigma}\left( g \right)$.
[/latex]
Back: [latex]
The bound is
\[
\mathbb{P} \left( \left| \nu_n^{\sigma}(g) \right|\ge K \sqrt{2t} \right) \le 2\exp \left[ -t \right].
\]
The only difference is the factor of $\sqrt{n} $, which we get by definition $\nu_n^{\sigma}(g) := \sqrt{n} \hat{P}_n ^{\sigma}g$.
[/latex]

Front: [latex]
Let $g$ be some function, $\hat{P}_n $ an empirical measure, and $\left\Vert \cdot \right\Vert _n$ the $L^{2} \left( \hat{P}_n \right)$ norm. Can we say $\hat{P}_n \left| g \right| \le \left\Vert g \right\Vert _n $?
[/latex]
Back: [latex]
Yes, we can. We can justify it with two alternative basic inequalities for probability measures. Let $X$ be a random variable:
\begin{enumerate}[(a)]
\item Since the function $x \mapsto x^{2}$ is convex, we can directly use Jensen's inequality to say $\left[ \Exnb X \right]^{2} \le \Exnb \left[ X^{2} \right]$.
\item We can use Cauchy-Schwarz on $X \cdot 1$ to write
\[\begin{aligned}
\left| \Exnb \left[ X \cdot 1 \right] \right|^{2}&\le \Exnb \left[ X^{2} \right]\Exnb \left[ 1^{2} \right] \\
\left| \Exnb \left[ X \right] \right|& \le \sqrt{\Exnb \left[ X^{2} \right]}.
\end{aligned}\]
\end{enumerate}
Both of the above directly translate into \[\Exnb_{\hat{P}_n }\left| g\left( X \right) \right| = \hat{P}_n \left| g \right| \le \underbrace{\sqrt{\hat{P}_n g^{2}}}_{=:\left\Vert g \right\Vert _n} = \sqrt{\Exnb _{\hat{P}_n }\left[ g^{2}\left( X \right) \right]}. \]
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $ with radius $R = \sup _{g \in \mathcal{G} } \left\Vert g \right\Vert$ and empirical radius $\hat{R} := \sup _{g \in \G} \left\Vert g \right\Vert _n $. Take a function $g \in \mathcal{G} $. Chaining allows us to represent it by telescoping as
\[\begin{aligned}
g &= g - g^{S + 1} + g^{S + 1} \\
&= g - g^{S + 1} + g^{S + 1} - g^{S} + g^{S} \\
&= g - g^{S + 1} + \sum_{s = 0}^{S} \left( g^{s + 1} - g^{s} \right).
\end{aligned}\]
For a given $s \in \left\{ 0, \dots, S \right\}$, use Hoeffding's inequality to find a bound in probability on
\[
\sup _{g \in \mathcal{G} } \left| \nu_n^{\sigma}\left( g^{s + 1} - g^{s} \right) \right|.
\]
[/latex]
Back: [latex]
We know that for each link in the chain with $s \in \left\{ 0, \dots, S \right\},$ we can bound its norm as $\left\Vert g^{s + 1} - g^{s} \right\Vert _n \le 2^{-s}\hat{R}_n .$ Furthermore, by Cauchy-Schwarz (or Jensen) we know that
\[\begin{aligned}
\left| \nu_n^{\sigma}\left( g \right) \right| = \left| \sqrt{n} \hat{P}_n^{\sigma} g \right| \le \left| \sqrt{n} \hat{P}_n \left| g \right| \right| \stackrel{\text{Cauchy-Schwarz} }{\le } \left| \sqrt{n} \sqrt{\hat{P}_n g^{2}} \right| = \sqrt{n} \left\Vert g \right\Vert _n.
\end{aligned}\]
So, we can use the $\left\Vert g^{s + 1} - g^{s} \right\Vert _n \le 2^{-s}\hat{R}_n $ bound for Hoeffding to get for all $g$
\[\begin{aligned}
\P \left( \left| \nu_n^{\sigma } \left( g^{s + 1} - g^{s} \right) \right| \ge 2 ^{-s}\hat{R}_n \sqrt{2 t} \right) \le 2\exp \left[ -t \right] \text{ for all $t$},
\end{aligned}\]
and now to correct for $\sup _{g \in \mathcal{G} }$ just add the entropy term $H_{s + 1} := \log N_{s + 1}$ to reach
\[\begin{aligned}
\P \left( \sup _{g} \left| \nu_n^{\sigma } \left( g^{s + 1} - g^{s} \right) \right| \ge 2 ^{-s}\hat{R}_n \sqrt{2 \left( t + H_{s + 1} \right)} \right) \le 2\exp \left[ -t \right] \text{ for all $t$}.
\end{aligned}\]
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $ with radius $R = \sup _{g \in \mathcal{G} } \left\Vert g \right\Vert$ and empirical radius $\hat{R} := \sup _{g \in \G} \left\Vert g \right\Vert _n $. Take a function $g \in \mathcal{G} $. Chaining allows us to represent it by telescoping as
\[\begin{aligned}
g &= g - g^{S + 1} + g^{S + 1} \\
&= g - g^{S + 1} + g^{S + 1} - g^{S} + g^{S} \\
&= g - g^{S + 1} + \sum_{s = 0}^{S} \left( g^{s + 1} - g^{s} \right).
\end{aligned}\]
For a given $s \in \left\{ 0, \dots, S \right\}$, how many functions $g^{s}$ are there?
[/latex]
Back: [latex]
There are $N_s := N\left( 2^{-s}\hat{R}_n , \mathcal{G} , \hat{P}_n \right)$ such functions.
[/latex]

Front: [latex]
For any random variables $X$, $Y$, and constants $\alpha = \alpha _1 + \alpha _2$ where $\alpha _1, \alpha _2 > 0$, what comparison operator goes in the middle?
\[\mathbb{P} \left( \left| X + Y \right| > \alpha \right) \overset{?}{\circ} \mathbb{P} \left( \left| X \right| > \alpha _1 \right) + \mathbb{P} \left( \left| Y \right| > \alpha _2 \right) \]
[/latex]
Back: [latex]
\[\mathbb{P} \left( \left| X + Y \right| > \alpha \right) \le \mathbb{P} \left( \left| X \right| > \alpha _1 \right) + \mathbb{P} \left( \left| Y \right| > \alpha _2 \right) \]
This is the "triangle inequality" for random variables.
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $ with radius $R = \sup _{g \in \mathcal{G} } \left\Vert g \right\Vert$ and empirical radius $\hat{R} := \sup _{g \in \G} \left\Vert g \right\Vert _n $. Take a function $g \in \mathcal{G} $. Chaining allows us to represent it by telescoping as
\[\begin{aligned}
g &= g - g^{S + 1} + g^{S + 1} \\
&= g - g^{S + 1} + g^{S + 1} - g^{S} + g^{S} \\
&= g - g^{S + 1} + \sum_{s = 0}^{S} \left( g^{s + 1} - g^{s} \right).
\end{aligned}\]
For each $s$, we can bound $\sup _g \left| \nu_n^{\sigma} \left( g^{s + 1} - g^{s} \right) \right|$ in probability with Hoeffding as
\[\begin{aligned}
\P \left( \sup _{g} \left| \nu_n^{\sigma } \left( g^{s + 1} - g^{s} \right) \right| \ge 2 ^{-s}\hat{R}_n \sqrt{2 \left( t + H_{s + 1} \right)} \right) \le 2\exp \left[ -t \right] \text{ for all $t$}.
\end{aligned}\]
Find a similar bound in probability for the sum over $s$ given as \[\sup _g \left| \sum_{s = 0}^{S} \nu_n^{\sigma }\left( g ^{s + 1}- g^{s} \right) \right|.\]
[/latex]
Back: [latex]
The bound is
\[
\P \left( \sup _g \left| \sum_{s = 0}^{S} \nu_n^{\sigma }\left( g ^{s + 1}- g^{s} \right) \right| \ge J_n + 4\hat{R}_n \sqrt{1 + t} \right) \le 2\exp \left[ -t \right]
\] where $J_n := \sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2H_{s + 1}}$.
\begin{enumerate}[1.]
\item To combine the bounds for $\nu_n^{\sigma}\left( g^{s + 1}- g^{s} \right)$ into a sum, we want to use the triangle inequality for probabilities which states for positive $\alpha _1, \alpha _2$ with $\alpha _1 +\alpha _2 = \alpha $ that
\[\mathbb{P} \left( \left| X + Y \right| > \alpha \right) \le \mathbb{P} \left( \left| X \right| > \alpha _1 \right) + \mathbb{P} \left( \left| Y \right| > \alpha _2 \right).\] So, we need to find appropriate $\alpha _s$ that we can plug into Hoeffding's inequality nicely.
\item
These will be
\[\begin{aligned}
\alpha _s &= 2^{-s}\hat{R}_n \sqrt{2 H_{s + 1}} + 2^{-s}\hat{R}_n \sqrt{2 \left( 1 + s \right)\left( 1 + t \right)} \\
&\ge 2^{-s}\hat{R}_n \sqrt{2 H_{s + 1} + 2\left( 1 + s \right)\left( 1 + t \right)}
\end{aligned}\]
which we can individually plug into the Hoeffding inequality given as a hint, except with $t$ replaced by $\left( 1 + s \right)\left( 1 + t \right)$.
\item Furthermore, we can nicely bound their sum
\[\begin{aligned}
\sum_{s = 0}^{S} \alpha _s &= \sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2H_{s + 1}} + \sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2 \left( 1 + s \right)\left( 1 + t \right)}\\
&= \underbrace{\sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2H_{s + 1}}}_{J_n} + \hat{R}_n \sqrt{1 + t} \underbrace{\sum_{s = 0}^{S} 2^{-s}\sqrt{2\left( 1 + s \right)}}_{\le 4} .
\end{aligned}\]
\item Hence, we get
\[\begin{aligned}
\P \left( \sup _g \left| \sum_{s = 0}^{S} \nu_n^{\sigma }\left( g ^{s + 1}- g^{s} \right) \right| \ge J_n + 4\hat{R}_n \sqrt{1 + t} \right) &\le \P \left( \sup _g \left| \sum_{s = 0}^{S} \nu_n^{\sigma }\left( g ^{s + 1}- g^{s} \right) \right| \ge \sum_{s = 0}^{S} \alpha _s \right) \\
&\stackrel{\text{Triangle} }{\le} \sum_{s = 0}^{S} \P \left( \sup _g \left| \nu_n^{\sigma} \left( g^{s + 1} - g^{s} \right) \right| \ge \alpha _s \right) \\
&\le 2\sum_{s = 0}^{S} \exp \left[ -\left( 1 + s \right)\left( 1 + t \right) \right] \le 2 \exp \left[ -t \right].
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $ with radius $R = \sup _{g \in \mathcal{G} } \left\Vert g \right\Vert$ and empirical radius $\hat{R}_n := \sup _{g \in \G} \left\Vert g \right\Vert _n $. Let $J_n = \sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2H_{s + 1}}$.
Show that for a symmetrized empirical process $\nu_n^{\sigma}$ it's true that
\[
\P \left( \sup _{g \in \G} \left| \nu_n^{\sigma }(g) \right| \ge J_n + 4\hat{R}_n \sqrt{1 + t} + \sqrt{n} 2^{-\left( S + 1 \right)}\hat{R}_n \right) \le 2\exp \left[ -t \right].
\]
[/latex]
Back: [latex]
We will use the chaining procedure with Hoeffding's inequality.
\begin{enumerate}[1.]
\item Recall that by the chaining we can represent $g$ as
\[\begin{aligned}
g &= g - g^{S + 1} + g^{S + 1} \\
&= g - g^{S + 1} + g^{S + 1} - g^{S} + g^{S} \\
&= g - g^{S + 1} + \sum_{s = 0}^{S} \left( g^{s + 1} - g^{s} \right).
\end{aligned}\] where we can bound the difference between each consecutive $g^s$. For the start of the chain we can say $\left\Vert g - g^{S + 1} \right\Vert _n \le 2^{-\left( S + 1 \right)}\hat{R}_n $, and for each subsequent element
\[
\left\Vert g^{s + 1} - g^{s} \right\Vert _n \le 2^{-s}\hat{R}_n .
\]
\item We can start off by dissecting $\left| \nu_n^{\sigma}(g) \right|$ with chaining as
\[\begin{aligned}
\left| \nu_n^{\sigma }(g) \right| &\le \left| \smash[b]{\underbrace{\sum_{s = 0}^{S} \nu_n^{\sigma } \left( g^{s + 1} - g^{s} \right)}_{\text{(a)} }} \right| + \left| \smash[b]{\underbrace{\nu_n^{\sigma} \left( g - g^{S + 1} \right) }_{\stackrel{\text{(b) }}{ \le \sqrt{n} 2^{-\left( S + 1 \right)}\hat{R}_n}}} \right|.
\end{aligned}\]
\vspace{0.3cm}
\item
Now we need to deal with bounding (a). For each $s$, by Hoeffding's inequality combined with union bound, we have
\[\begin{aligned}
\P \left( \sup _{g} \left| \nu_n^{\sigma } \left( g^{s + 1} - g^{s} \right) \right| \ge 2 ^{-s}\hat{R}_n \sqrt{2 \left( t + H_{s + 1} \right)} \right) \le 2\exp \left[ -t \right] \text{ for all $t$}.
\end{aligned}\]
\item To combine the bounds for $\nu_n^{\sigma}\left( g^{s + 1}- g^{s} \right)$ into a sum, we want to use the triangle inequality for probabilities which states for positive $\alpha _1, \alpha _2$ with $\alpha _1 +\alpha _2 = \alpha $ that
\[\mathbb{P} \left( \left| X + Y \right| > \alpha \right) \le \mathbb{P} \left( \left| X \right| > \alpha _1 \right) + \mathbb{P} \left( \left| Y \right| > \alpha _2 \right).\] So, we need to find appropriate $\alpha _s$ that we can plug into Hoeffding's inequality nicely.
\item
These will be
\[\begin{aligned}
\alpha _s &= 2^{-s}\hat{R}_n \sqrt{2 H_{s + 1}} + 2^{-s}\hat{R}_n \sqrt{2 \left( 1 + s \right)\left( 1 + t \right)} \\
&\ge 2^{-s}\hat{R}_n \sqrt{2 H_{s + 1} + 2\left( 1 + s \right)\left( 1 + t \right)}
\end{aligned}\]
which we can individually plug into the Hoeffding inequality we found in step 3., except with $t$ replaced by $2\left( 1 + s \right)\left( 1 + t \right)$.
\item Furthermore, we can nicely bound their sum
\[\begin{aligned}
\sum_{s = 0}^{S} \alpha _s &= \sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2H_{s + 1}} + \sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2 \left( 1 + s \right)\left( 1 + t \right)}\\
&= \underbrace{\sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2H_{s + 1}}}_{=:J_n} + \hat{R}_n \sqrt{1 + t} \underbrace{\sum_{s = 0}^{S} 2^{-s}\sqrt{2\left( 1 + s \right)}}_{\le 4} .
\end{aligned}\]
\item
Hence, using the above with the triangle inequality,
\[\begin{aligned}
\P \left( \sup _g \left| \sum_{s = 0}^{S} \nu_n^{\sigma }\left( g ^{s + 1}- g^{s} \right) \right| \ge J_n + 4\hat{R}_n \sqrt{1 + t} \right) &\le \P \left( \sup _g \left| \sum_{s = 0}^{S} \nu_n^{\sigma }\left( g ^{s + 1}- g^{s} \right) \right| \ge \sum_{s = 0}^{S} \alpha _s \right) \\
&\stackrel{\text{Triangle} }{\le} \sum_{s = 0}^{S} \P \left( \sup _g \left| \nu_n \left( g^{s + 1} - g^{s} \right) \right| \ge \alpha _s \right) \\
&\le 2\sum_{s = 0}^{S} \exp \left[ -\left( 1 + s \right)\left( 1 + t \right) \right] \\
&\le 2 \exp \left[ -t \right].
\end{aligned}\]
\item Now we have all the pieces to complete the proof. Specifically, the steps
\begin{enumerate}[(a)]
\item (Step 2.) $ \left| \nu_n^{\sigma }(g) \right| \le \left| \sum_{s = 0}^{S} \nu_n^{\sigma } \left( g^{s + 1} - g^{s} \right) \right| + \sqrt{n} 2^{-\left( S + 1 \right)}\hat{R}_n$
\item (Step 7.) $ \P \left( \sup _g \left| \sum_{s = 0}^{S} \nu_n^{\sigma }\left( g ^{s + 1}- g^{s} \right) \right| \ge J_n + 4\hat{R}_n \sqrt{1 + t} \right) \le 2 \exp \left[ -t \right]$
\end{enumerate}
result in the conclusion
\[\begin{aligned}
\P \left( \sup _{g \in \G} \left| \nu_n^{\sigma }(g) \right| \ge J_n + 4\hat{R}_n \sqrt{1 + t} + \sqrt{n} 2^{-\left( S + 1 \right)}\hat{R}_n \right) &\le 2\exp \left[ -t \right].
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $ with radius $R = \sup _{g \in \mathcal{G} } \left\Vert g \right\Vert$ and empirical radius $\hat{R}_n := \sup _{g \in \G} \left\Vert g \right\Vert _n $. As a result of the chaining procedure we can write the inequality below for a symmetrized empirical process $\nu_n^{\sigma}$:
\[
\P \left( \sup _{g \in \G} \left| \nu_n^{\sigma }(g) \right| \ge J_n + 4\hat{R}_n \sqrt{1 + t} + \sqrt{n} 2^{-\left( S + 1 \right)}\hat{R}_n \right) \le 2\exp \left[ -t \right]
\]
What is the formula for term $J_n$?
[/latex]
Back: [latex]
\[J_n = \sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2H_{s + 1}}\]
[/latex]

Front: [latex]
Suppose for a class of functions $\mathcal{G} $ with radius $R = \sup _{g \in \mathcal{G} } \left\Vert g \right\Vert$ there exists a decreasing function $H\left( \cdot \right)$ such that $H_2\left( u, \mathcal{G} , \mathcal{Q} \right) \le H(u)$ for all $u$ and for all probability measures $\mathcal{Q} $. Define $H_s := H_2 \left( 2^{-s}\hat{R}_n, \G, \hat{P}_n \right)$. Find Dudley's entropy integral bound for
\[J_n = \sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2H_{s + 1}}.\]
[/latex]
Back: [latex]
We will treat the sum of $J_n$ as an integral -- a sum of areas of rectangles under a curve -- and bound it appropriately.
\begin{enumerate}[1.]
\item We want to write $J_n$ as a sum of areas of disjoint rectangles. We can do so by
\[\begin{aligned}
J_n := \sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2H_{s + 1}} &= \sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2H_2\left( 2^{-\left( s + 1 \right)} \hat{R}_n , \mathcal{G} , \hat{P}_n \right)}\\
&=\sum_{s = 0}^{S} \left( 2^{-(s-1)} - 2^{-s} \right)\hat{R}_n \sqrt{2H_2\left( 2^{-\left( s + 1 \right)} \hat{R}_n , \mathcal{G} , \hat{P}_n \right)}\\
\end{aligned}\]
\item The above form already allows us to write a nice integral. However, since $H_2\left( \cdot, \mathcal{G} , \hat{P}_n \right)$ is decreasing, we want the rightmost endpoint of each rectangle to have the height equal to the value of $H_2\left( \cdot, \mathcal{G} , \hat{P}_n \right)$ at that point. For this purpose, we need to "squish" the rectangles by shifting the right endpoint from $2^{-\left( s - 1 \right)}$ to $2^{-\left( s + 1 \right)}$ by taking $2^{2}$ out of the sum, as below. See the figure for a visual explanation for the final bounds for some example entropy function $H_2$ and bounding function $H$.
\[\begin{aligned}
J_n &=\sum_{s = 0}^{S} \left( 2^{-(s-1)} - 2^{-s} \right)\hat{R}_n \sqrt{2H_2\left( 2^{-\left( s + 1 \right)} \hat{R}_n , \mathcal{G} , \hat{P}_n \right)} \\
&= 4\sum_{s = 0}^{S} \left( 2^{-(s + 1)} - 2^{-(s + 2)} \right)\hat{R}_n \sqrt{2H_2\left( 2^{-\left( s + 1 \right)} \hat{R}_n , \mathcal{G} , \hat{P}_n \right)} &\text{(Squish by factor of $2^{2}$)} \\
&= 4\sum_{s = 0}^{S} \int_{2^{-(s + 2)}\hat{R}_n }^{2^{-(s + 1)}\hat{R}_n } \sqrt{2H_2\left( 2^{-\left( s + 1 \right)} \hat{R}_n , \mathcal{G} , \hat{P}_n \right)}du\\
&\le 4\sum_{s = 0}^{S} \int_{2^{-(s + 2)}\hat{R}_n }^{2^{-(s + 1)}\hat{R}_n } \sqrt{2H_2\left( u, \mathcal{G} , \hat{P}_n \right)}du &\text{($u \mapsto H_2\left( u, \mathcal{G} , \hat{P}_n \right)$ decreasing)} \\
&\le 4\sum_{s = 0}^{S} \int_{2^{-(s + 2)}\hat{R}_n }^{2^{-(s + 1)}\hat{R}_n } \sqrt{2H\left( u \right)}du &\text{(Assumption)} \\
&\le 4 \int_{2^{-(S + 2)}\hat{R}_n }^{2^{-1}\hat{R}_n } \sqrt{2H\left( u \right)}du. \\
\end{aligned}\]
\begin{center}\includegraphics[width=0.8\textwidth]{ept_jn_uniform_entropy_helper.png}\end{center}
\item We can end with a nicer looking form of the bound as
\[\begin{aligned}
J_n &\le 4 \int_{2^{-(S + 2)}\hat{R}_n }^{2^{-1}\hat{R}_n } \sqrt{2H\left( u \right)}du \\
&\le 4 \int_{0}^{\hat{R}_n } \sqrt{2H\left( u \right)du.}
\end{aligned}\]
\end{enumerate}
[The bound given in the notes has a factor of 2. I cannot explain why.]
[/latex]

Front: [latex]
Suppose $\mathcal{G} $ is a class of functions with envelope $G$. What is the uniform entropy condition for $\mathcal{G} $?
[/latex]
Back: [latex]
There exists a decreasing function $\mathcal{H} $ satisfying $\int_{0}^{1} \sqrt{\mathcal{H} \left( u \right)} du < \infty$ such that for all probability measures $Q $ and for all $u$ we have $H_2\left( u \sqrt{\mathcal{Q} G^{2}} , \mathcal{G} , Q \right) \le \mathcal{H} \left( u \right).$
[/latex]

Front: [latex]
Suppose $\mathcal{G} $ is a class of functions with radius $R = \sup _{g \in \mathcal{G} } \left\Vert g \right\Vert$. What does the uniform entropy condition on $\mathcal{G} $ imply about the sum $J_n := \sum_{s = 0}^{S} 2^{-s}\hat{R}_n \sqrt{2H_{s + 1}}$ of entropies from chaining?
[/latex]
Back: [latex]
If the uniform entropy condition holds, so there exists a decreasing function $\mathcal{H} $ satisfying $\int_{0}^{1} \sqrt{\mathcal{H} \left( u \right)} du < \infty$ such that for all probability measures $Q $ and for all $u$ we have $H_2\left( u , \mathcal{G} , Q \right) \le \mathcal{H} \left( u \right),$ then
\[\begin{aligned}
J_n &\le \int_{2^{-(S + 1)} \hat{R}_n }^{\hat{R}_n } \sqrt{2 \mathcal{H} \left( u \right)} du \\
&\le \int_{0}^{\hat{R}_n } \sqrt{2 \mathcal{H} \left( u \right)} du.
\end{aligned}\]
[/latex]

Front: [latex]
Take some $g \in \mathcal{G} $. Suppose for the class of functions $\mathcal{G} $ the uniform entropy condition holds, and their envelope $G$ is in $L^{2} \left( P \right)$.
Show that the above conditions imply $\nu_n^{\sigma}\left( g \right)$ is asymptotically continuous.
[/latex]
Back: [latex]
We want to show that the conditions for asymptotic continuity hold, so
\begin{enumerate}[(1)]
\item $G \in L_2(P)$,
\item $\frac{1}{n} \log N_2 \left( \delta , \G, \hat{P}_n \right) \xrightarrow[n \to \infty]{\P} 0$, and
\item $\nu_n^{\sigma }\left( g - g_0 \right) \xrightarrow[]{\P} 0$ as $\left\Vert g - g_0 \right\Vert_n \to 0$,
\end{enumerate}
Conditions (1) and (2) follow directly from the uniform entropy condition. The difficult bit is (3). Here, we want to show that for all $\varepsilon > 0$, there exists a $\delta > 0$ such that if the indexes are at empirical distance at most $\delta $ with high probability, then the function values are at distance at most $\varepsilon $ with high probability.
\begin{enumerate}[1.]
\item Without loss of generality suppose that $G \equiv 1$. Take $\varepsilon > 0$. Let $\mathcal{G} \left( \delta \right) := \left\{ g \in \mathcal{G} : \left\Vert g - g_0 \right\Vert_n \le \delta \right\}$ be the set of functions which are at empirical distance at most $\delta $.
\item Define $\mathcal{J} \left( \delta \right) := \int_{0}^{\delta } \sqrt{2\mathcal{H} \left( u \right)} du$.
\item Consider the inequality we know to be true with probability $1 - 2\exp \left[ -t \right]$
\[
\sup _{g \in \mathcal{G} } \left| \nu_n^{\sigma} \right| \le \underbrace{J_n + 4\hat{R}_n \sqrt{1 + t}}_{\text{(a)} } + \underbrace{\sqrt{n} 2^{-\left( S + 1 \right)}\hat{R}_n}_{\text{(b)} } .
\]
Note that the empirical radius $\hat{R}_n $ is now $\delta $ on the subset $\mathcal{G} \left( \delta \right)$ we are considering. So, $\hat{R}_n $ is finite and we can dismiss the term (b) as it must converge to $0$ as $S \to \infty$. Now we want to get the terms in (a) to 0.
\item Since $\mathcal{J} \left( \delta \right) = \int_{0}^{\delta } \sqrt{2H\left( u \right)} du $ is a convergent integral, we can take a $\delta $ to make it as small as we want. So, take
\begin{itemize}
\item $t$ such that $2\exp \left[ -t \right]\le \varepsilon $, and
\item $\delta$ such that $\mathcal{J} \left( \delta \right)\le \frac{\varepsilon }{2}$ and $4\delta \sqrt{1 + t} \le \frac{\varepsilon }{2}$.
\end{itemize}
\item Now we have with probability $1 - \varepsilon $ that
\[
\sup _{\stackrel{g \in \mathcal{G} }{\left\Vert g - g_0 \right\Vert_{n} \le \delta } }\left| \nu_n^{\sigma}\left( g - g_0 \right) \right| \le \varepsilon,
\] so we showed $\hat{\nu}_n^{\sigma} \left( g - g_0 \right) \xrightarrow[]{\mathbb{P} } 0$ as $\left\Vert g - g_0 \right\Vert _n \to 0$.
\item Now we can combine the above with the remaining conditions for asymptotic continuity to conclude that $\nu_n^{\sigma}\left( g \right)$ is asymptotically continuous.
\end{enumerate}
This is theorem 9.4.1.
[/latex]

Front: [latex]
Take some $g \in \mathcal{G} $. Suppose for the class of functions $\mathcal{G} $ the uniform entropy condition holds, and their envelope $G$ is in $L^{2} \left( P \right)$. What does this imply about $\nu_n^{\sigma}\left( g \right)$?
[/latex]
Back: [latex]
These imply that $\nu_n^{\sigma}$ is asymptotically continuous.
This is theorem 9.4.1.
[/latex]

Front: [latex]
Consider two sequences $\left( a_{n} \right)_{n \geq 1} $ and $\left( b_{n} \right)_{n \geq 1} $. What does it mean if $a_n \lesssim b_n$?
[/latex]
Back: [latex]
$a_n \lesssim b_n$ if there exists a universal constant $C$ such that for all $n$,
\[
a_n < Cb_n.
\]
[/latex]

Front: [latex]
Consider a function space $\mathcal{G} $ and a probability measure $Q$ on it. Consider an $L^{1} \left( Q \right)$-norm and an $L^{2} \left( Q \right)$-norm ball on $\mathcal{G} $ with the same radius. Which of these balls contains more points? What does this imply about the covering numbers $N_1\left( \delta , \mathcal{G} , Q \right)$ and $N_2\left( \delta , \mathcal{G} , Q \right)$?
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item For any probability measure $Q$ and functions $g, \tilde{g}\in \mathcal{G} $ we have
\[
\left\Vert g - \tilde{g} \right\Vert _{L^{1} \left( Q \right)} = Q\left| g - \tilde{g} \right| \stackrel{\text{Cauchy-Schwarz} }{\le } \sqrt{Q\left| g - \tilde{g} \right|^{2}} = \left\Vert g - \tilde{g} \right\Vert _{L^{2} \left( Q \right)}.
\]
Hence, the distance between two functions is greater in $L^{2} \left( Q \right)$ than in $L^{1} \left( Q \right)$. So, a ball of same radius covers fewer points in $L^{2} \left( Q \right)$ than in $L^{1} \left( Q \right)$.
\item
This means that the $L^{2} \left( Q \right)$ covering number is larger for the same radius $\delta $
\[
N_2\left( \delta , \mathcal{G} , Q \right) \ge N_1\left( \delta , \mathcal{G} , Q \right).
\]
\end{enumerate}
[/latex]

Front: [latex]
Suppose $\mathcal{G} $ is a Vapnik-Chervonenkis. Show that then the uniform entropy condition holds.
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item By theorem 6.4.1 we know since $\mathcal{G} $ is Vapnik-Chervonenkis it's true that the covering number $N_1$ behaves like a polynomial in $1/\delta $. For all $\delta > 0$ and $\overline{Q}$,
\[
N_1 \left( \delta \overline{Q}G, \mathcal{G} , \overline{Q} \right) \le A \delta ^{-V}\log ^{V}\left( \frac{1}{\delta } \right).
\] The right hand side is integrable from 0 to 1, so we want to now turn the $L^{1} $ norm into the $L^{2} $ norm to show
\[
H_2 \left( \delta \sqrt{Q G^{2}} , \mathcal{G} , Q \right) \le \mathcal{H} \left( \delta \right)
\] where $\mathcal{H} \left( \delta \right)$ will come from the term $ A \delta ^{-V}\log ^{V}\left( \frac{1}{\delta } \right)$.
\item For any probability measure $Q$ and functions $g, \tilde{g}\in \mathcal{G} $ we have
\[
\left\Vert g - \tilde{g} \right\Vert _{L^{1} \left( Q \right)} = Q\left| g - \tilde{g} \right| \stackrel{\text{Cauchy-Schwarz} }{\le } \sqrt{Q\left| g - \tilde{g} \right|^{2}} = \left\Vert g - \tilde{g} \right\Vert _{L^{2} \left( Q \right)}.
\]
Hence, the distance between two functions is greater in $L^{2} \left( Q \right)$ than in $L^{1} \left( Q \right)$. So, a ball of same radius covers fewer points in $L^{2} \left( Q \right)$ than in $L^{1} \left( Q \right)$. We will thus try to construct an alternative probability space $\overline{Q}$ where we can lower bound the $L^{1} \left( \overline{Q} \right)$ distances by $L^{2} \left( Q \right)$.
% Below is the proof based on the lecture, but it's probably wrong. She bounds the L_1(\bar{Q}) norm from below, but we actually want to bound it from above. That we can do simply by Cauchy-Schwarz
\item So, consider the $L^{2} \left( Q \right)$ distance between two functions $g$ and $\tilde{g}$. We will use the envelope to go from a squared $L^{2} $ norm to an $L^{1} $ norm as
\[\begin{aligned}
\overbrace{Q\left( g - \tilde{g} \right)^{2}}^{L^{2} \left( Q \right)\text{-norm squared} } &= Q\left| g - \tilde{g} \right| \left| g - \tilde{g} \right| \\
&\le Q \left| g - \tilde{g} \right|2G \\
&= \tilde{Q}\left| g - \tilde{g} \right| &(d\tilde{Q} := 2GdQ) \\
&= \underbrace{\overline{Q} \left| g - \tilde{g} \right|}_{L^{1} \left( \overline{Q} \right)\text{-norm} } \cdot \int_{}^{} 2GdQ &\left( \text{$\overline{Q}$ is a prob. measure, so $d\overline{Q} := \frac{2GdQ}{\int_{}^{} 2GdQ}$} \right)
\end{aligned}\]
\item By the above we can write that for any $g$, $\tilde{g}$
\[
\left\Vert g - \tilde{g} \right\Vert _{L^{2} \left( Q \right)} = K \implies \left\Vert g - \tilde{g} \right\Vert _{L^{1} \left( \overline{Q} \right) } \ge \frac{K^{2}}{2QG}.
\] So, a ball of radius $K$ in $L^{2} \left( Q \right)$ contains at most as many points as a ball of radius $K^{2}/2QG$ in $L^{1} \left( \overline{Q} \right)$.
\item Now, plug the above into $N_2\left( \delta \sqrt{QG^{2}} , \mathcal{G} , Q \right)$ as
\[\begin{aligned}
N_2\left(\smash[b]{ \underbrace{\delta \sqrt{Q\left( 2G \right)^{2}}}_{=:K} , \mathcal{G} , Q } \right) &\le N_1 \left( \frac{\delta ^{2}Q\left( 2G \right)^{2} }{2QG}, \mathcal{G} , \overline{Q} \right)\\
&= N_1\left( \delta ^{2}\overline{Q}\left( 2G \right), \mathcal{G} , \overline{Q} \right)\\
&= N_1\left( \delta'\overline{Q}G, \mathcal{G} , \overline{Q} \right) \\
&\le A'\delta ^{-2V}\log ^{V}\left( \frac{1}{\delta ^{2}} \right) &\text{ for some constant $A'$}.
\end{aligned}\]
\item So, we know that the entropy $H_2\left( u\sqrt{QG^{2}} , \mathcal{G} , Q \right)$ is indeed bounded by $\mathcal{H} \left( u \right)$ with $\int_{0}^{1} \sqrt{\mathcal{H} \left( u \right)du} < \infty$, thus fulfilling the uniform entropy condition for $\mathcal{G} $.
\end{enumerate}
[/latex]

Front: [latex]
Suppose a class $ g\in \mathcal{G} $ is Vapnik-Chervonenkis with an envelope $G \in L^{2} \left( P \right)$. What does this mean for an empirical process $\nu_n$ on $\mathcal{G} $?
[/latex]
Back: [latex]
This means that $\nu_n$ is asymptotically continuous.
\begin{enumerate}[1.]
\item We know that $\mathcal{G} $ being Vapnik-Chervonenkis implies that the uniform entropy condition holds for $\mathcal{G} $.
\item We know that the uniform entropy condition together with the envelope $G$ satisfying $G \in L^{2} \left( P \right)$ implies that $\nu_n^{\sigma}$ is asymptotically continuous.
\item Since $\nu_n^{\sigma}$ is asymptotically continuous, so is $\nu_n$.
\end{enumerate}
[/latex]

Front: [latex]
Suppose for some M-estimator $\hat{\theta }_n $ we managed to show asymptotic normality
\[
\sqrt{n} \left( \hat{\theta }_n - \theta _0 \right) \xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, \Sigma \right).
\] What transformation must we apply to the above for the limiting distribution to be $\mathcal{N} \left( 0, I \right)$?
[/latex]
Back: [latex]
We need to transform it by $\Sigma ^{-1/2}$
\[
\sqrt{n} \Sigma ^{-1/2}\left( \hat{\theta }_n - \theta _0 \right) \xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, I \right).
\]
[/latex]

Front: [latex]
Suppose for some M-estimator $\hat{\theta }_n $ we managed to show asymptotic normality
\[
\sqrt{n} \left( \hat{\theta }_n - \theta _0 \right) \xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, \Sigma \right).
\] Suppose also that $\hat{\Sigma }_n $ is a consistent estimator of $\Sigma $. How can you construct an approximate $\left( 1 - \alpha \right)$-confidence set for $\theta $?
[/latex]
Back: [latex]
The approximate $\left( 1 - \alpha \right)$-confidence set for $\theta $ is
\[
\left\{ \theta : n\left( \hat{\theta }_n - \theta \right)^{T}\hat{\Sigma }_n ^{-1}\left( \hat{\theta }_n - \theta \right) \le q\left( 1 - \alpha \right) \right\}
\] where $q\left( 1 - \alpha \right)$ is the $\left( 1-\alpha \right)$-quantile of $\chi^{2}_r $.
[/latex]

Front: [latex]
What does it mean for an M-estimator $\hat{\theta }_n $ to be asymptotically linear?
[/latex]
Back: [latex]
$\hat{\theta }_n $ is called asymptotically linear if
\[
\sqrt{n} \left( \hat{\theta }_n - \theta_0 \right) = \sqrt{n} \hat{P}_n l + o_\mathbb{P} \left( 1 \right)
\] where
\begin{enumerate}[1.]
\item $l : \mathcal{X} \to \mathbb{R} ^{r}$ is the influence function, $l = \begin{pmatrix}
l_1 \\
\vdots \\
l_r
\end{pmatrix}$, $\Exnb l(X) = 0$, and
\item
$\Sigma := P ll^{T} < \infty$ is the asymptotic covariance matrix.
\end{enumerate}
[/latex]

Front: [latex]
What does asymptotic linearity have to do with asymptotic normality?
[/latex]
Back: [latex]
If an estimator $\hat{\theta }_n $ is asymptotically linear, then it is also asymptotically normal.
[/latex]

Front: [latex]
Suppose an M-estimator $\hat{\theta }_n $ is consistent ($\hat{\theta }_n \xrightarrow[n \to \infty]{\mathbb{P} } \theta _0$), the parameter space $\Theta $ is finite-dimensional, and the true parameter $\theta _0$ is an interior point of $\Theta $.
What are the (a), (b), (c) (strong) conditions for asymptotic normality of $\hat{\theta }_n $?
[/latex]
Back: [latex]
\begin{enumerate}[(a)]
\item Loss functions $\gamma _\theta $ are differentiable in a neighbourhood of $\theta _0$. There exists an $\varepsilon > 0$ such that if $\left| \theta - \theta _0 \right| \le \varepsilon $ then for all $x \in \mathcal{X} $ the partial derivative exists \[\psi_\theta \left( x \right) := \frac{\partial}{\partial \theta } \gamma _\theta \left( x \right).\]
\item The mean of this derivative is itself differentiable. As $\theta \to \theta _0$,
\[
P\left( \psi _\theta - \psi _{\theta _0} \right) = \left(V + o \left( 1 \right) \right) \left( \theta - \theta _0 \right), V \in \mathbb{R} ^{r \times r}.
\] where the Hessian $V \in \mathbb{R} ^{r \times r}$ is a positive definite matrix.
\item There exists an $\varepsilon > 0$ such that the empirical process $\nu_n$ indexed by the class of derivatives of the loss in a neighbourhood of $\theta _0$ \[\left\{ \psi _\theta : \left| \theta - \theta _0 \right| \le \varepsilon \right\}\] is asymptotically continuous at $\psi _{\theta _0}$. Furthermore,
\[
\lim_{\theta \to \theta _0} \left\Vert \psi _\theta - \psi _{\theta _0} \right\Vert_{L^{2} \left( P \right)} = 0.
\]
\end{enumerate}
[/latex]

Front: [latex]
Consider a class of functions $\mathcal{G} $ parametrized by $\theta $, and suppose $\theta \mapsto g_\theta (x)$ is continuous for all $x$. What property of $\mathcal{G} $ can allow you to say
\[\left(\hat{P}_n-P\right)\left(g_{\theta } - g_{\theta _0}\right) = o_{\textbf{p} }\left( n^{-1/2} \right)\]
as $\theta \to \theta _0$?
[/latex]
Back: [latex]
This just means that the empirical process $\nu_n$ indexed by $\mathcal{G} $ is asymptotically continuous at $g_{\theta _0}$.
[/latex]

Front: [latex]
Suppose an M-estimator $\hat{\theta }_n $ is consistent ($\hat{\theta }_n \xrightarrow[n \to \infty]{\mathbb{P} } \theta _0$), the parameter space $\Theta $ is finite-dimensional, and the true parameter $\theta _0$ is an interior point of $\Theta $. Assume the (strong) a, b, c conditions for asymptotic normality hold.
Show that then $\hat{\theta }_n $ is asymptotically linear with influence function
\[
l = -V^{-1}\psi _{\theta _0},
\] meaning that
\[
\sqrt{n} \left( \hat{\theta }_n - \theta _0 \right)\xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, V^{-1}JV^{-1} \right)
\] where $J = P\psi _{\theta _0}\psi _{\theta _0}^{T}$.
Recall the a, b, c conditions:
\begin{enumerate}[(a)]
\item There exists an $\varepsilon > 0$ s.t. $ \psi _\theta \left( x \right) := \frac{\partial}{\partial \theta } \gamma _\theta \left( x \right)$ exists for all $\left| \theta - \theta _0 \right| \le \varepsilon $.
\item As $\theta \to \theta _0$ we have $P\left( \psi _\theta - \psi _{\theta _0} \right) = \left( V + o\left( 1 \right) \right)\left( \theta - \theta _0 \right)$ where $V \in \mathbb{R} ^{r \times r}$ is positive definite.
\item There exists an $\varepsilon > 0$ s.t. $\nu_n$ is an empirical process indexed by $\left\{ \psi _\theta : \left| \theta - \theta _0 \right| \le \varepsilon \right\}$ and $\nu_n$ is asymptotically continuous at $\psi _{\theta _0}$. Also, $\lim_{\theta \to \theta _0} \left\Vert \psi _\theta - \psi _{\theta _0} \right\Vert_2 = 0.$
\end{enumerate}
[/latex]
Back: [latex]
The key is to use and rewrite the score equation $\hat{P}_n \psi _{\hat{\theta }_n } = 0$ in such a way that we can insert condition (b) into it.
\begin{enumerate}[1.]
\item Recall that $\theta _0$ is an interior point of $\Theta $, and minimizes $P\gamma _\theta $, so that $P\psi _{\theta _0} = 0$. Because $\hat{\theta }_n $ is consistent, it is eventually a solution of the score equation
\[
\hat{P}_n \psi _{\hat{\theta }_n } = 0.
\]
\item
Rewrite the score equation as
$$
\begin{aligned}
0 &=\hat{P}_n \psi_{\hat{\theta}_{n}}=\hat{P}_n\left(\psi_{\hat{\theta}_{n}}-\psi_{\theta_{0}}\right)+\hat{P}_n \psi_{\theta_{0}} \\
&=\left(\hat{P}_n-P\right)\left(\psi_{\hat{\theta}_{n}}-\psi_{\theta_{0}}\right)+P\left( \psi_{\hat{\theta}_{n}} - \psi _{\theta _0}\right)+\hat{P}_n \psi_{\theta_{0}} \\
&=o_{\mathbf{P}}\left(n^{-1 / 2}\right)+P\left( \psi_{\hat{\theta}_{n}} - \psi _{\theta _0}\right) +\hat{P}_n \psi_{\theta_{0}} &\text{(c)}\\
&=o_{\mathbf{P}}\left(n^{-1 / 2}\right)+ V\left(\hat{\theta}_{n}-\theta_{0}\right)+o\left(\left|\hat{\theta}_{n}-\theta_{0}\right|\right) +\hat{P}_n \psi_{\theta_{0}} &\text{(b)}
\end{aligned}
$$
\item
Rearranging terms, we get the desired
$$
\left(\hat{\theta}_{n}-\theta_{0}\right)=-V^{-1} \hat{P}_n \psi_{\theta_{0}}+o_{\mathbf{P}}\left(n^{-1 / 2}\right) .
$$
\end{enumerate}
[/latex]

Front: [latex]
Give the statement of condition (b) for asymptotic normality. How is the matrix $V$ related to $\frac{\partial}{\partial \theta } P\psi _\theta $?
[/latex]
Back: [latex]
The condition states that as $\theta \to \theta _0$ we have $P\left( \psi _\theta - \psi _{\theta _0} \right) = \left( V + o\left( 1 \right) \right)\left( \theta - \theta _0 \right)$ where $V \in \mathbb{R} ^{r \times r}$ is positive definite.
The $V$ matrix can be interpreted as the first order Taylor expansion of $P\psi _\theta $ at $\theta _0$, i.e. \[V = \lim_{\theta \to \theta _0}V_\theta = \lim_{\theta \to \theta _0} \frac{\partial}{\partial \theta } P\psi _\theta.\]
[/latex]

Front: [latex]
Consider the problem of estimating the parameter $\theta $ of a logistic distribution by MLE. Let $X \in \mathbb{R} $ with
\begin{center}\begin{tabular}{ c | c | c }
distribution function & density function & negative log likelihood loss \\[10pt]
$P\left( X \le x \right) = \frac{1}{1 + e^{\theta _0 - x}}$ & $p_{\theta _0}(x) = \frac{1}{\left( 1 + e^{\theta _0 - x} \right)^{2}} e^{\theta _0 - x}$ & $\gamma _\theta \left( x \right) = -\left( \theta - x - 2\log \left( 1 + e^{\theta - x} \right) \right)$
\end{tabular}\end{center}
Show that $\sqrt{n} \left( \hat{\theta }_n - \theta _0 \right)$ is asymptotically linear, and find its asymptotic variance.
[/latex]
Back: [latex]
We need to check the conditions a, b, c for asymptotic normality.
\begin{enumerate}[(a)]
\item The first condition asks that the derivative exists in a neighbourhood of $\theta _0$.
\[\begin{aligned}
\frac{\partial}{\partial \theta } \gamma _\theta (x) = \psi _\theta (x) &= -1 + \frac{2 }{1 + e^{\theta - x}} e^{\theta - x} \\
&= -1 + 2 \left( 1 - \frac{1}{1 + e^{\theta - x}} \right) \\
&= 1 - \frac{2}{1 + e^{\theta - x}}.
\end{aligned}\]
\item The second condition says that we take the mean of the above derivative, and then we get something that is differentiable again. In this case, the above function is bounded by 1, so we can interchange integration and differentiation. So, first take the derivative, and then take the integral at $\theta = \theta _0$.
\[\begin{aligned}
\frac{\partial}{\partial \theta } \psi _\theta \left( x \right) |_{\theta = \theta _0} = \frac{2}{\left( 1 + e^{\theta - x} \right)^{2}}e^{\theta - x} |_{\theta = \theta _0} =: \dot{\psi }_{\theta _0}.
\end{aligned}\]
\[\begin{aligned}
\Exnb _{\theta _0}\dot{\psi }_{\theta _0} \left( X \right) &= \int_{-\infty}^{\infty} \frac{2}{\left( 1 + e^{\theta_0 - x} \right)^{2}}e^{\theta _0 - x}p_{\theta _0}(x) dx \\
&= \int_{}^{} \frac{2}{\left( 1 + e^{\theta_0 - x} \right)^{2}}e^{\theta _0 - x} d\left( \frac{1}{1 + e^{\theta _0 - x}} \right) &\left( p_{\theta _0}(x) = \frac{\partial}{\partial x} \frac{1}{1 + e^{\theta _0 - x}} \right) \\
&=2\int_{}^{} \frac{e^{\theta _0}}{1 + e^{\theta _0 - x}} \cdot \frac{1}{1 + e^{\theta _0 - x}} d\left( \frac{1}{1 + e^{\theta _0 - x}} \right)\\
&=2 \int_{0}^{1} \frac{\left( 1 - u \right)}{u}du \\
&= 2\left[ \frac{1}{2}u^{2} - \frac{1}{3}u^{3} \right|_0^{1} = \frac{1}{3}.\\
&\implies V = \frac{1}{3}.
\end{aligned}\]
\item We want to check for asymptotic continuity. We can show it by showing that the class of graphs of loss function derivatives $\psi _\theta $ is Vapnik-Chervonenkis. We have
\[
\left\{ \left( x, t \right) : \psi _\theta \left( x \right) \ge t \right\} = \left\{ \left( x, t \right) : x \le \log \left( \frac{1 - t}{1 + t} \right)+\theta \right\}
\] by rearranging the terms so that $\theta $ and $t$ are on one side, $x$ on the other, and taking the log. So, we can see that the graph is equivalent to a half space. Hence, the class of functions
\[
\left\{ \left\{ \left( x, y \right) : x \le \log \left( \frac{1 - t}{1 + t} \right)+\theta \right\} : \theta \in \mathbb{R} \right\}.
\] is Vapnik-Chervonenkis with envelope $\sup _{x} \left| \psi _\theta \left( x \right) \right| \le 1$. Thus, by dominated convergence $\left( \theta \mapsto \psi _\theta \left( x \right) \text{ is continuous} \right)$ we get
\[
\lim_{\theta \to \theta _0} \left\Vert \psi _\theta -\psi _{\theta _0} \right\Vert = 0.
\]
\end{enumerate}
So, based on these conditions, we conclude that
\[
\sqrt{n} \left( \hat{\theta }_n - \theta _0 \right) \xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, \Sigma \right)
\] where the limiting variance is $\Sigma = V^{-1}JV^{-1} = 3$, since $V = \frac{1}{3}$, $J = V$.
[/latex]

Front: [latex]
Suppose $\Theta \subset \mathbb{R} $ and we have two estimators of the same quantity which are both asymptotically normal
\[\begin{aligned}
\sqrt{n} \left( \hat{\theta }_{n,1} - \theta _0 \right) &\xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, \sigma ^{2}_1 \right) \text{, }
\sqrt{n} \left( \hat{\theta }_{n,2} - \theta _0 \right) &\xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, \sigma ^{2}_2 \right).
\end{aligned}\] What is asymptotic relative efficiency $e_{1, 2}$? When can you say that the estimator $\hat{\theta }_{n, 1}$ is more efficient than $\hat{\theta }_{n, 2}$? How do you interpret $e_{1, 2}$ in the context of statistical inference?
[/latex]
Back: [latex]
Asymptotic relative efficiency is $e_{1,2} = \frac{\sigma ^{2}_2}{\sigma ^{2}_1}$. If $e_{1, 2} > 1$ , then the first estimator is more efficient than the second one.
The value of $e_{1, 2}$ is relevant when we want to construct a confidence interval with a certain length. If we use the second estimator instead of the first, we need approximately $e_{1, 2}$ times as many observations to reach the same length.
[/latex]

Front: [latex]
Consider the problem of logistic regression in $\mathbb{R} ^{r}$. Let $\left( X, Y \right) \in \mathbb{R} ^{r}\times \left\{ 0, 1 \right\}$, $\eta (x) := P\left( Y = 1 | X = x \right)$, $\Sigma _X := \Exnb X^{T}X < \infty$.
\begin{center}\begin{tabular}{ c | c | c }
conditional probability & model & loss function \\[10pt]
$\eta (x) := P\left( Y = 1 | X = x \right)$ & $\log \frac{\eta (x)}{1 - \eta (x)} \stackrel{\text{model} }{=} x\theta _0$ & $\gamma _\theta(x, y) = -yx\theta + \log \left( 1 + e^{x\theta } \right)$
\end{tabular}\end{center}
where $x$ is a row vector, $\theta _0$ a column vector.
Show that $\hat{\theta }_n $ is asymptotically linear and state its covariance matrix in terms of $\eta $ and $X$.
[/latex]
Back: [latex]
We want to check the a, b, c conditions for asymptotic normality:
\begin{enumerate}[(a)]
\item The derivative of the loss function exists
\[\begin{aligned}
\psi _\theta \left( x, y \right) &= -yx^{T} + \frac{1}{1 + e^{x\theta }} e^{x\theta }x^{T}.
\end{aligned}\]
\item Does the second derivative exist? We will do the trick with swapping the order of integration: first take the derivative, then the mean of the derivative at $\theta _0$
\[\begin{aligned}
\dot{\psi }_\theta \left( x, y \right)& = \frac{-1}{\left( 1 + e^{x\theta } \right)^{2}} e^{2x\theta }x^{T}x + \frac{1}{1 + e^{x\theta }}e^{x\theta } x^{T}x\\
\implies \dot{\psi }_{\theta _0} \left( x, y \right) &= \eta (x) \left( 1 - \eta (x) \right) x^{T}x.
\end{aligned}\] Now
\[\begin{aligned}
V &= \Exnb _{\theta _0}\eta \left( X \right) \left( 1 - \eta (X) \right)X^{T}X.
\end{aligned}\]
\item The class of graphs of the loss function derivatives
\[
\left\{ \left\{ \left( x, y, t \right) : \psi _\theta \left( x, y \right) \ge t \right\} :\theta \in \mathbb{R} ^{r} \right\}
\] is Vapnik-Chervonenkis. To find the envelope, see that
\[\begin{aligned}
\left| \psi _\theta \left( X, Y \right) \right| &= \left| -YX^{T} + \frac{1}{1 + e^{X\theta }} e^{X\theta }X^{T} \right| \\
&\le \left| \smash[b]{ -\underbrace{Y}_{\in \left\{ 0, 1 \right\}}X^{T}} \right| + \left| \smash[b]{\underbrace{\frac{1}{1 + e^{X\theta }} e^{X\theta }}_{\left| \cdot \right|\le 1}X^{T} } \right|\\[15pt]
&\le 2\left| X \right| \in L^{2} \left( P \right).
\end{aligned}\]
\end{enumerate}
Hence, since the a, b, c conditions for asymptotic normality hold, we get
\[
\sqrt{n} \left( \hat{\theta }_n - \theta _0 \right) \xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, V^{-1} \right).
\]
[/latex]

Front: [latex]
Suppose an M-estimator $\hat{\theta }_n $ is consistent ($\hat{\theta }_n \xrightarrow[n \to \infty]{\mathbb{P} } \theta _0$), the parameter space $\Theta $ is finite-dimensional, and the true parameter $\theta _0$ is an interior point of $\Theta $.
What are the (aa), (bb), (cc) (weak) conditions for asymptotic normality of $\hat{\theta }_n $?
[/latex]
Back: [latex]
The conditions are:
\begin{enumerate}[label=(\alph*\alph*)]
\item (Differentiability in quadratic mean) There exists a function $\psi _0 : \mathcal{X} \to \mathbb{R} ^{r}$ with $P\psi _0 = 0$ and $P\psi _{0}\psi _0^{T} =: J < \infty$, $k = 1, \dots, r$ such that
\[
\lim_{\theta \to \theta _0} \frac{\left\Vert \gamma _\theta - \gamma _{\theta _0} - \left( \theta - \theta _0 \right)^{T}\psi _0 \right\Vert_{L^{2} \left( P \right)} }{\left| \theta - \theta _0 \right|} = 0.
\]
\item We have as $\theta \to \theta _0$,
\[
P\left( \gamma _\theta - \gamma _{\theta _0} \right) = \frac{1}{2}\left( \theta - \theta _0 \right)^{T} V\left( \theta - \theta _0 \right) + o\left( 1 \right)\left| \theta - \theta _0 \right|^{2},
\] with $V \in \mathbb{R} ^{r \times r}$ a positive definite matrix.
\item Define the functions
\[
g_\theta =
\begin{cases}
\frac{\gamma _\theta - \gamma _{\theta _0} - \left( \theta - \theta _0 \right)^{T}\psi _0}{\left| \theta - \theta _0 \right|} & \theta \neq \theta _0,\\
0&\theta = \theta _0.
\end{cases}
\] Require that for some $\varepsilon > 0$ the class $\mathcal{G} := \left\{ g_\theta : 0 < \left| \theta - \theta _0 \right| < \varepsilon \right\}$ has $\nu_n : \mathcal{G} \to \mathbb{R} $ asymptotically continuous at 0.
\end{enumerate}
[/latex]

Front: [latex]
The (weak) condition (bb) for asymptotic normality of a consistent M-estimator $\hat{\theta }_n $ is
\begin{enumerate}[(bb)]
\item We have as $\theta \to \theta _0$,
\[
P\left( \gamma _\theta - \gamma _{\theta _0} \right) = \frac{1}{2}\left( \theta - \theta _0 \right)^{T} V\left( \theta - \theta _0 \right) + o\left( 1 \right)\left| \theta - \theta _0 \right|^{2},
\] with $V \in \mathbb{R} ^{r \times r}$ a positive definite matrix.
\end{enumerate}
We can consider the above to be a second order Taylor expansion of $\gamma _\theta $ at $\theta _0$, with $V$ as the quadratic term. Why are there no linear terms in the expansion?
[/latex]
Back: [latex]
The linear term is 0 because the theoretical target parameter is the minimizer of theoretical risk, so it will be a stationary point. $\left( \left( \frac{\partial}{\partial \theta } P\gamma _\theta \right) |_{\theta =\theta _0} = 0 \right)$
[/latex]

Front: [latex]
The (weak) condition (cc) for asymptotic normality of a consistent M-estimator $\hat{\theta }_n $ is
\begin{enumerate}[(cc)]
\item Define the functions
\[
g_\theta =
\begin{cases}
\frac{\gamma _\theta - \gamma _{\theta _0} - \left( \theta - \theta _0 \right)^{T}\psi _0}{\left| \theta - \theta _0 \right|} & \theta \neq \theta _0,\\
0&\theta = \theta _0.
\end{cases}
\] Require that for some $\varepsilon > 0$ the class $\mathcal{G} := \left\{ g_\theta : 0 < \left| \theta - \theta _0 \right| < \varepsilon \right\}$ has $\nu_n : \mathcal{G} \to \mathbb{R} $ asymptotically continuous at 0.
\end{enumerate}
What is the standard approach for checking this condition?
[/latex]
Back: [latex]
We typically check this condition by evaluating whether the class $\mathcal{G} $ is Vapnik-Chervonenkis, so by looking into whether the class of its graphs is Vapnik-Chervonenkis.
[/latex]

Front: [latex]
Suppose an M-estimator $\hat{\theta }_n $ is consistent ($\hat{\theta }_n \xrightarrow[n \to \infty]{\mathbb{P} } \theta _0$), the parameter space $\Theta $ is finite-dimensional, and the true parameter $\theta _0$ is an interior point of $\Theta $. Assume the (weak) aa, bb, cc conditions for asymptotic normality hold.
Show that then $\hat{\theta }_n $ is asymptotically linear with influence function
\[
l = -V^{-1}\psi _{\theta _0},
\] meaning that
\[
\sqrt{n} \left( \hat{\theta }_n - \theta _0 \right)\xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, V^{-1}JV^{-1} \right)
\] where $J = P\psi _{\theta _0}\psi _{\theta _0}^{T}$.
Recall the aa, bb, cc conditions:
\begin{enumerate}[label=(\alph*\alph*)]
\item (Differentiability in quadratic mean) There exists a function $\psi _0 : \mathcal{X} \to \mathbb{R} ^{r}$ with $P\psi _0 = 0$ and $P\psi _{0}\psi _0^{T} =: J < \infty$, $k = 1, \dots, r$ such that
\[
\lim_{\theta \to \theta _0} \frac{\left\Vert \gamma _\theta - \gamma _{\theta _0} - \left( \theta - \theta _0 \right)^{T}\psi _0 \right\Vert_{L^{2} \left( P \right)} }{\left| \theta - \theta _0 \right|} = 0.
\]
\item We have as $\theta \to \theta _0$,
\[
P\left( \gamma _\theta - \gamma _{\theta _0} \right) = \frac{1}{2}\left( \theta - \theta _0 \right)^{T} V\left( \theta - \theta _0 \right) + o\left( 1 \right)\left| \theta - \theta _0 \right|^{2},
\] with $V \in \mathbb{R} ^{r \times r}$ a positive definite matrix.
\item Define the functions
\[
g_\theta =
\begin{cases}
\frac{\gamma _\theta - \gamma _{\theta _0} - \left( \theta - \theta _0 \right)^{T}\psi _0}{\left| \theta - \theta _0 \right|} & \theta \neq \theta _0,\\
0&\theta = \theta _0.
\end{cases}
\] Require that for some $\varepsilon > 0$ the class $\mathcal{G} := \left\{ g_\theta : 0 < \left| \theta - \theta _0 \right| < \varepsilon \right\}$ has $\nu_n : \mathcal{G} \to \mathbb{R} $ asymptotically continuous at 0.
\end{enumerate}
[/latex]
Back: [latex]
We will perform calculations for $V = I$, and finally replace $\left( \theta - \theta _0 \right)$ by $V^{1/2}\left( \theta - \theta _0 \right)$.
\begin{enumerate}[1.]
\item
We start by looking at the empirical loss of the estimator and compare it to the empirical loss of the truth. Since $\hat{\theta }_n $ is the minimizer of $\hat{P}_n \gamma _\theta $, we can start with
\[\begin{aligned}
0 &\ge \hat{P}_n \left( \gamma _{\hat{\theta }_n } - \gamma _{\theta _0} \right)\\
&= \left( \hat{P}_n - P \right)\left( \gamma _{\hat{\theta }_n } - \gamma _{\theta _0} \right) + P\left( \gamma _{\hat{\theta }_n} - \gamma _{\theta _0} \right) \\
&=\underbrace{\left( \hat{P}_n - P \right)\frac{\gamma _{\hat{\theta }_n } - \gamma _{\theta _0} - \left( \hat{\theta }_n - \theta \right) \psi _0}{ \left| \theta - \theta _0 \right|}}_{(\star)} \left| \theta - \theta _0 \right| \\
&+ \left( \hat{\theta }_n - \theta _0 \right)\underbrace{\left( \hat{P}_n - P \right)\psi _0}_{=\mathcal{O} _\mathcal{P} \left( \frac{1}{\sqrt{n} } \right)\text{ by CLT} } + \underbrace{P\left( \gamma _{\hat{\theta }_n } - \gamma _{\theta _0} \right)}_{\left( \dagger \right)}\\
\end{aligned}\]
\item
Consider the term $\left( \star \right)$.
\begin{enumerate}[(1)]
\item By condition aa $\frac{\gamma _{\hat{\theta }_n } - \gamma _{\theta _0} - \left( \hat{\theta }_n - \theta \right) \psi _0}{ \left| \theta - \theta _0 \right|} \xrightarrow[n \to \infty]{L^{2} \left( P \right)} 0$.
\item Condition cc states that the empirical process
\[
\sqrt{n} \left( \hat{P}_n - P \right)\frac{\gamma _{\hat{\theta }_n } - \gamma _{\theta _0} - \left( \hat{\theta }_n - \theta \right) \psi _0}{ \left| \theta - \theta _0 \right|}
\] is asymptotically continuous at 0, which combined with (1) implies that it converges to 0.
\item So, we can write
\[
\left( \hat{P}_n - P \right)\frac{\gamma _{\hat{\theta }_n } - \gamma _{\theta _0} - \left( \hat{\theta }_n - \theta \right) \psi _0}{ \left| \theta - \theta _0 \right|} = o_\mathbb{P} \left( \frac{1}{\sqrt{n} } \right).
\]
\end{enumerate}
\item By condition bb, we can replace the term $\left( \dagger \right)$ by
\[
\frac{1}{2}\left( \theta - \theta _0 \right)^{T} V \left( \theta - \theta _0 \right) + o\left( \left| \theta - \theta _0 \right| \right) \stackrel{V := I}{=}
\frac{1}{2}\left( \theta - \theta _0 \right)^{2}+ o\left( \left| \theta - \theta _0 \right| \right)^{2}
\]
\item Coming back to the big inequality, we get
\[\begin{aligned}
0 &\ge \underbrace{o_{\mathbb{P} }\left( \frac{1}{\sqrt{n} } \right) \left| \theta - \theta _0 \right|}_{\left( \heartsuit.1 \right)} + \underbrace{\left( \hat{\theta }_n - \theta _0 \right)\mathcal{O} _\mathcal{P}\left( \frac{1}{\sqrt{n} } \right)}_{\left( \heartsuit.2 \right)} + \underbrace{\frac{1}{2}\left( \theta - \theta _0 \right)^{2}}_{\left( \heartsuit.3 \right)}+ \underbrace{o\left( \left| \theta - \theta _0 \right| \right)^{2}}_{\left( \heartsuit.4 \right)}.
\end{aligned}\]
\item We will now try to "complete the square" with the above. Consider how the terms from the inequality above connect with terms in the quadratic form below
\[\begin{aligned}
&\frac{1}{2}\left( \left( \hat{\theta }_n - \theta _0 \right)\left( 1 + o\left( 1 \right) \right) + \mathcal{O} _\mathbb{P} \left( \frac{1}{\sqrt{n} } \right) \right)^{2 }\\
=&\underbrace{ \frac{1}{2}\left( \hat{\theta }_n - \theta _0 \right)^{2}}_{= \left( \heartsuit.3 \right)} + \underbrace{\left( \hat{\theta }_n - \theta _0 \right) \mathcal{O} _\mathbb{P} \left( \frac{1}{\sqrt{n} } \right)}_{= \left( \heartsuit.2 \right)} + \mathcal{O} _\mathbb{P} \left( \frac{1}{n} \right)\\
&+ \underbrace{\frac{1}{2}o\left( \hat{\theta }_n - \theta _0 \right)^{2}}_{= \left( \heartsuit.4 \right)} + \underbrace{o\left( \hat{\theta }_n - \theta _0 \right) \mathcal{O} _\mathbb{P} \left( \frac{1}{\sqrt{n} } \right)}_{= \left( \heartsuit.1 \right)}.
\end{aligned}\]
\item So, we can write
\[\begin{aligned}
0 &\ge \frac{1}{2}\left( \left( \hat{\theta }_n - \theta _0 \right)\left( 1 + o\left( 1 \right) \right) + \mathcal{O} _\mathbb{P} \left( \frac{1}{\sqrt{n} } \right) \right)^{2 } + \mathcal{O} _\mathbb{P} \left( \frac{1}{n} \right).
\end{aligned}\]
\item This inequality means that
\[\begin{aligned}
\left( \left( \hat{\theta }_n - \theta _0 \right)\left( 1 + o\left( 1 \right) \right) + \mathcal{O} _\mathbb{P} \left( \frac{1}{\sqrt{n} } \right) \right)^{2 } &= \mathcal{O} _\mathbb{P} \left( \frac{1}{n} \right)\\
\implies \left( \hat{\theta }_n - \theta _0 \right)\left( 1 + o\left( 1 \right) \right) + \mathcal{O} _\mathbb{P} \left( \frac{1}{\sqrt{n} } \right) &= \mathcal{O} _\mathbb{P} \left( \frac{1}{\sqrt{n} } \right) \\
\implies \left( \hat{\theta }_n - \theta _0 \right) &= \mathcal{O} _\mathbb{P} \left( \frac{1}{\sqrt{n} } \right).
\end{aligned}\]
\item
Define $\tilde{\theta }_n$ with $\left( \tilde{\theta }_n - \theta _0 \right) = -\hat{P}_n \psi _0$. By CLT, we have $\left( \tilde{\theta }_n - \theta _0 \right) = \mathcal{O} _\mathbb{P} \left( \frac{1}{\sqrt{n} } \right).$ Now perform the same expansion as in the large inequality on $\hat{P}_n \left( \gamma _{\tilde{\theta }_n} - \gamma _{\theta _0} \right)$ to get
\[\begin{aligned}
\hat{P}_n \left( \gamma _{\hat{\theta }_n} - \gamma _{\theta _0} \right)= \left( \tilde{\theta }_n - \theta _0 \right)^{T}\hat{P}_n \psi _0 + \frac{1}{2}\left( \tilde{\theta }_n - \theta _0 \right)^{2} + o_\mathbb{P} \left( \frac{1}{n} \right).
\end{aligned}\]
Since MLE is the minimizer, we know $\begin{aligned}
\hat{P}_n \left( \gamma _{\tilde{\theta }_n} - \gamma _{\theta _0} \right) \ge \hat{P}_n \left( \gamma _{\hat{\theta }_n} - \gamma _{\theta _0} \right).
\end{aligned}$ So write
\[\begin{aligned}
\hat{P}_n \left( \gamma _{\tilde{\theta }_n} - \gamma _{\theta _0} \right)&= \left( \hat{\theta }_n - \theta _0 \right)^{T}\hat{P}_n \psi _0 + \frac{1}{2}\left( \hat{\theta }_n - \theta _0 \right)^{2} + o_\mathbb{P} \left( \frac{1}{n} \right) \\
&\le -\frac{1}{2}\left( \hat{P}_n \psi _0 \right)^{2} + o_{\mathbb{P} }\left( \frac{1}{n} \right). &\text{($\tilde{\theta }_n - \theta _0 := -\hat{P}_n \psi _0$)}
\end{aligned}\]
\item We can use the above to find the order of $\left( \hat{\theta }_n - \theta _0 \right)$ as
\[\begin{aligned}
\frac{1}{2}\left( \left( \hat{\theta }_n - \theta _0 \right) + \hat{P}_n \psi _0 \right)^{2} &= o_{\mathbb{P} } \left( \frac{1}{n} \right)\\
\implies \left( \hat{\theta }_n - \theta _0 \right) + \hat{P}_n \psi _0 &= o_{\mathbb{P} }\left( \frac{1}{\sqrt{n} } \right)
\end{aligned}\]
so
\[\begin{aligned}
\sqrt{n} \left( \hat{\theta }_n - \theta _0 \right) &= -\sqrt{n} \hat{P}_n \psi _0 + o_\mathbb{P} \left( 1 \right)\\
&\xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, P\psi _0\psi _0^{T} \right). &\text{(CLT)}
\end{aligned}\]
\item
Putting the $V$ back in, i.e. replacing $\left( \hat{\theta }_n - \theta _0 \right)$ by $V^{1/2}\left( \hat{\theta }_n - \theta _0 \right)$, we get the desired
\[
\sqrt{n} \left( \hat{\theta }_n - \theta _0 \right) = -\sqrt{n} \hat{P}_n V^{-1}\psi _0 + o_\mathbb{P} \left( 1 \right).
\] with influence function $l = -V^{-1}\psi _0$ and convergence to $\mathcal{N} \left( 0, V^{-1}JV^{-1} \right)$.
\end{enumerate}
[/latex]

Front: [latex]
Consider the problem of finding the median. Let $X \in \mathbb{R} $, with
\begin{center}\begin{tabular}{ c | c | c | c }
distribution & density & target & loss function \\[10pt]
$F(x)$ & $f = F'$ & $\theta _0 = F^{-1}\left( \frac{1}{2} \right)$ & $\gamma _\theta \left( x \right) = \left| x - \theta \right|$
\end{tabular}\end{center}
Show that the M-estimator $\hat{\theta }_n $ is asymptotically linear and state its variance in terms of the density $f$.
[/latex]
Back: [latex]
The loss functions is not differentiable at 0, so we can't use the strong conditions a, b, c for asymptotic normality. So, we wish to verify the weak conditions aa, bb, cc.
\begin{enumerate}[label=(\alph*\alph*)]
\item The derivative of the loss, where it is defined, is
\[
\psi _0\left( x \right) =
\begin{cases}
1 &\text{ if } \theta _0 > x, \\
-1 &\text{ if } \theta _0 < x.
\end{cases}
\]
At 0 it is undefined; for the purposes of integration, it can have any value, since it's just a single point. Take the case $\theta > \theta _0$. The alternative case will be very similar, just with flipped signs at the end.
\[\begin{aligned}
\gamma _\theta \left( x \right) - \gamma _{\theta _0} \left( x \right) - \left( \theta - \theta _0 \right)\psi _0\left( x \right) &= \left| x - \theta \right| - \left| x - \theta _0 \right| - \left( \theta - \theta _0 \right)\psi _0\left( x \right) \\
&=
\begin{cases}
\theta - x - \left( \theta _0 - x \right) - \left( \theta - \theta _0 \right) &\text{ if }x < \theta _0 < \theta ,\\
\theta - x - \left( x - \theta _0 \right) + \left( \theta - \theta _0 \right) &\text{ if } \theta _0 < x < \theta ,\\
x - \theta - \left( x - \theta _0 \right) + \left( \theta - \theta _0 \right) &\text{ if } \theta _0 < \theta < x,
\end{cases}\\
&=
\begin{cases}
0 &\text{ if }x < \theta _0 < \theta ,\\
-2 \left( x - \theta \right) &\text{ if } \theta _0 < x < \theta ,\\
0 &\text{ if } \theta _0 < \theta < x.
\end{cases}
\end{aligned}\]
Now consider the differentiability in $L^{2} $ statement
\[\begin{aligned}
\frac{\left\Vert \gamma _\theta - \gamma _{\theta _0} - \left( \theta - \theta _0 \right) \psi _0 \right\Vert ^{2}}{\left| \theta - \theta _0 \right|^{2}} &= \int_{\substack{\theta_0 < x < \theta \\ \theta < x < \theta_0} }^{} \underbrace{\frac{4 \left( x - \theta \right)^{2}}{\left| \theta - \theta _0 \right|^{2}}}_{\left| x - \theta \right| < \left| \theta - \theta _0 \right|}f(x) dx\\
&\le 4 \int_{ \substack{\theta_0 < x < \theta \\ \theta < x < \theta_0} }^{} f(x)dx \\
&= 4 \left( F\left( \theta \right) - F\left( \theta _0 \right) \right)\\
&\xrightarrow[\theta \to \theta _0]{} 0.
\end{aligned}\]
So, differentiability in quadratic mean is fine.
\item
$\gamma _\theta $ is not twice differentiable, so we can't first take the derivative and then integrate. So, first integrate, and then take the derivative.
\[\begin{aligned}
P\gamma _\theta &= \int_{x < \theta }^{} \left( \theta - x \right)dF\left( x \right) + \int_{x > \theta }^{} \left( x - \theta \right) dF\left( x \right)\\
&= \int_{x < 0}^{} F(x)dx + \int_{x > \theta }^{} \left( 1 - F(x) \right)dx. &\text{(Partial integration)}
\end{aligned}\]
Now take derivative
\[\begin{aligned}
\frac{\partial}{\partial \theta } P\gamma _\theta &= F(\theta ) - \left( 1 - F(\theta ) \right) \\
&=2F\left( \theta \right) - 1 .
\end{aligned}\]
which we see vanishes as $\theta = \theta _0$, and take the second
\[\begin{aligned}
\frac{\partial^{2}}{\partial\theta ^{2}} P\gamma _\theta |_{\theta = \theta _0} = 2f \left( \theta _0 \right) = V.
\end{aligned}\]
\item We need to show asymptotic continuity. Take the class $\mathcal{G} $
\[\begin{aligned}
\left\{ \left\{ x \mapsto \frac{\left| x - \theta \right| - \left| x - \theta _0 \right| - \left( \theta - \theta _0 \right)\psi _0\left( x \right)}{\left| \theta - \theta _0 \right|} \right\} : \theta \in \mathbb{R} \right\}
\end{aligned}\]
We can see that the functions in the numerator are Lipschitz, hence Vapnik-Chervonenkis, and also bounded by 2 after division by $\left| \theta - \theta _0 \right|$. So, we have asymptotic continuity.
\end{enumerate}
Now to conclude, compute $J$ and $\Sigma $ as
\[\begin{aligned}
J &= P\psi _0^{2} = 1\\
\Sigma &= V^{-1}JV^{-1} = \frac{1}{4f\left( \theta _0 \right)^{2}},
\end{aligned}\]
and we get
\[
\sqrt{n} \left( \hat{\theta }_n - \theta _0 \right) \xrightarrow[n \to \infty]{\mathcal{L} } \mathcal{N} \left( 0, \frac{1}{4f\left( \theta _0 \right)^{2}} \right).
\]
[/latex]

Front: [latex]
Consider an M-estimator $\hat{\theta }_n $. Show that
\[\begin{aligned}
0 &\le P\left( \gamma_{\hat{\theta}_n} - \gamma_{\theta_0} \right) \\
&\le - \left( \hat{P}_n - P \right)\left( \gamma_{\hat{\theta}_n} - \gamma_{\theta_0} \right).
\end{aligned}\]
[/latex]
Back: [latex]
\[\begin{aligned}
0 &\le P\left( \gamma_{\hat{\theta}_n} - \gamma_{\theta_0} \right)\\
&=- \left( \hat{P}_n - P \right) \left( \gamma_{\hat{\theta}_n} - \gamma_{\theta_0} \right) + \underbrace{\hat{P}_n \left( \gamma_{\hat{\theta}_n} - \gamma_{\theta_0} \right)}_{\le 0}
\end{aligned}\]
[/latex]

Front: [latex]
In context of M-estimators, what are the excess risk and the basic inequality?
[/latex]
Back: [latex]
Excess risk is the difference between the optimal theoretical risk and the theoretical risk of the estimator $P\left( \gamma _{\hat{\theta }_n } - \gamma _{\theta _0} \right)$.
The basic inequality states that the excess risk can be bounded by some empirical process
\[\begin{aligned}
0 &\le P\left( \gamma_{\hat{\theta}_n} - \gamma_{\theta_0} \right)\\
&\le - \left( \hat{P}_n - P \right)\left( \gamma_{\hat{\theta}_n} - \gamma_{\theta_0} \right)
\end{aligned}\]
[/latex]

Front: [latex]
Prove the basic inequality for least squares regression
\[
\left\Vert \hat{g}_n - g ^{0} \right\Vert_n^{2} \le 2\hat{\varepsilon }_n ^{T}\left( \hat{g}_n - g ^{0} \right)/n
\] where the $x_i$ are fixed, $Y_i = g_0\left( x_i \right) + \varepsilon _i$, $\varepsilon _1, \dots, \varepsilon _n$ are i.i.d. $\mathcal{N} \left( 0, \sigma \right)$, $\hat{\varepsilon }_n $ are the empirical errors, the norm $\left\Vert v \right\Vert _n^{2} := \frac{1}{n}\sum_{i = 1}^{n} v_i^{2}$, quadratic loss $\gamma _g\left( x, y \right) = \left( y - g\left( x \right) \right)^{2}$.
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item First write out $\hat{P}_n \gamma _{\hat{g}_n }$ for any $g$ fully
\[\begin{aligned}
\hat{P}_n \gamma _g = \left\Vert Y - g \right\Vert _n^{2} &= \left\Vert \varepsilon -\left( g - g^{0} \right) \right\Vert _n^{2} \\
&=\left\Vert \varepsilon \right\Vert ^{2} - 2\hat{\varepsilon }_n ^{T}\left( g - g^{0} \right)/n + \left\Vert g -g^{0} \right\Vert _n^{2}
\end{aligned}\]
\item Now write out the terms in $0 \ge \hat{P}_n \gamma _{\hat{g}_n } - \hat{P}_n \gamma _{g^{0} }$
\[\begin{aligned}
0 &\ge \hat{P}_n \gamma _{\hat{g}_n } - \hat{P}_n \gamma _{g^{0} } \\
&= \cancel{\left\Vert \varepsilon \right\Vert ^{2}} - 2\hat{\varepsilon }_n ^{T}\left( \hat{g}_n - g^{0} \right)/n + \left\Vert \hat{g}_n - g ^{0} \right\Vert ^{2}_n\\
&- \cancel{\left\Vert \varepsilon \right\Vert ^{2}} - \cancel{2\hat{\varepsilon }_n ^{T}\left( g^{0} - g^{0} \right)/n} + \cancel{\left\Vert g^{0} - g ^{0} \right\Vert ^{2}_n}\\
&= - 2\hat{\varepsilon }_n ^{T}\left( \hat{g}_n - g^{0} \right)/n + \left\Vert \hat{g}_n - g ^{0} \right\Vert ^{2}_n
\end{aligned}\] and conclude
\[
2\hat{\varepsilon }_n ^{T}\left( \hat{g}_n - g^{0} \right)/n \ge \left\Vert \hat{g}_n - g ^{0} \right\Vert ^{2}_n.
\]
\end{enumerate}
[/latex]

Front: [latex]
In the context of least squares estimation, what does $\mathcal{G} \left( \delta \right)$ mean for $\delta > 0$?
[/latex]
Back: [latex]
This is the class of functions in the vicinity of the target $g^{0} $
\[
\mathcal{G} \left( \delta \right) := \left\{ g \in \mathcal{G} : \left\Vert g - g^{0} \right\Vert _n \le \delta \right\}.
\]
[/latex]

Front: [latex]
State the entropy integrability condition and describe it intuitively.
[/latex]
Back: [latex]
The condition is
\[
J\left( \delta \right) := 2\int_{0}^{\delta } \sqrt{2H\left( u, \mathcal{G} \left( \delta \right), \left\Vert \cdot \right\Vert _n \right)} du < \infty. \text{ for all }\delta > 0.
\]
It states that the Euclidean space entropy of a $\delta$-ball $\mathcal{G} \left( \delta \right)$ around the target does not grow too fast as the covering radius $u$ goes to 0.
[/latex]

Front: [latex]
Consider the problem of least squares regression. Let $Y_i = g\left( x_i \right) + \varepsilon _i$ where $x_i$ are fixed, $\varepsilon _1, \dots, \varepsilon _n$ are i.i.d. $\mathcal{N} \left( 0, \sigma \right)$, $\hat{\varepsilon }_n $ a vector of their empirical values. Define $J\left( \delta \right) := 2\int_{0}^{\delta } \sqrt{2H\left( u, \mathcal{G} \left( \delta \right), \left\Vert \cdot \right\Vert _n \right)} du$.
Justify why it is true that for any $\delta > 0$ and for all $t > 0$, we have
\[\begin{aligned}
\mathbb{P} \left( \sup _{g \in \mathcal{G} \left( \delta \right)} \hat{\varepsilon }_n \left( g - g^{0} \right)/n \ge \frac{J\left( \delta \right)}{ \sqrt{n} } + 4\delta \sqrt{\frac{1 + t}{n}} \right) \le \exp \left[ -t \right]. &\text{ } (*)
\end{aligned}\]
You may refer to the result derived for a symmetrized empirical process $\nu_n^{\sigma}$:
\[
\P \left( \sup _{g \in \G} \left| \nu_n^{\sigma }(g) \right| \ge J_n + 4\hat{R}_n \sqrt{1 + t} + \sqrt{n} 2^{-\left( S + 1 \right)}\hat{R}_n \right) \le 2\exp \left[ -t \right].
\]
[/latex]
Back: [latex]
The given inequality can be proven analogously to the one for symmetrized empirical processes $\nu_n^{\sigma}$, which was shown using chaining. The inequality for $\nu_n^{\sigma}$ relied on Hoeffding's inequality and the fact that the Rademacher random variables used in symmetrization are bounded. There is a similar Hoeffding-esque inequality for normally distributed random variables which we can use in the proof for $\left( * \right)$.
[/latex]

Front: [latex]
Consider the problem of least squares regression. Let $Y_i = g\left( x_i \right) + \varepsilon _i$ where $x_i$ are fixed, $\varepsilon _1, \dots, \varepsilon _n$ are i.i.d. $\mathcal{N} \left( 0, \sigma \right)$, $\hat{\varepsilon }_n $ a vector of their empirical values.
Show that if
\begin{enumerate}[(i)]
\item for all $\delta > 0$ the integral
\[J\left( \delta \right) := 2\int_{0}^{\delta } \sqrt{2H\left( u, \mathcal{G} \left( \delta \right), \left\Vert \cdot \right\Vert _n \right)} du\]
is finite,
\item $\frac{J\left( \delta \right)}{\delta ^{2} }$ is decreasing in $\delta $, and
\item for all $t > 0$ \[\delta _n^{2} \ge 8 \left( \frac{J\left( \delta _n \right)}{\sqrt{n} } + 4\delta _n \sqrt{\frac{1 + t}{n}} \right)\]
\end{enumerate}
then for all $t > 0 $ it holds that
\[
\mathbb{P} \left( \left\Vert \hat{g}_n - g ^{0} \right\Vert_n > \delta _n \right) \le \frac{1}{e-1}\exp \left[ -t \right].
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item We would like to use the chaining result which states that for any $\delta > 0$ and for all $t > 0$, we have
\[\begin{aligned}
\mathbb{P} \left( \sup _{g \in \mathcal{G} \left( \delta \right)} \hat{\varepsilon }_n \left( g - g^{0} \right)/n \ge \smash[b]{\underbrace{\frac{J\left( \delta \right)}{ \sqrt{n} } + 4\delta \sqrt{\frac{1 + t}{n}}}_{\left( \dagger \right)}} \right) \le \exp \left[ -t \right].
\end{aligned}\] \vspace{0.1cm}
We will use "peeling", so we will show that $\left\Vert \hat{g}_n - g ^{0} \right\Vert $ is, with high probability, not in donut shaped subsets of $\mathcal{G} $
\[
\left\{ g \in \mathcal{G} : 2^{j - 1}\delta _n < \left\Vert g - g_0 \right\Vert \le 2^{j}\delta _n \right\}
\] for progressively larger $j$ starting from a radius of $\delta _n$. This will allow us to write
\[
\mathbb{P} \left( \left\Vert \hat{g}_n - g^{0} \right\Vert _n > \delta _n \right) \le \sum_{j = 1}^{\infty} \mathbb{P} \left( \sup _{g \in \mathcal{G} \left( 2^{j}\delta _n \right)} 2 \hat{\varepsilon }_n \left( g - g^{0} \right)/n \ge \left( 2^{j - 1} \delta _n \right)^{2} \right).
\]
\item Use the assumption that $\frac{J\left( \delta \right)}{\delta ^{2}}$ is decreasing in $\delta $ to put an upper bound on $\left( \dagger \right)$ depending on $\delta _n$ and $j$
\[\begin{aligned}
& \frac{J\left( 2^{j}\delta _n \right)+4\cdot 2^{j}\delta _n \sqrt{1 + t + j}}{\left( 2^{j}\delta _n \right)^{2}} \\
\le& \frac{J\left( \delta _n \right) + 4 \delta _n \sqrt{1 + t} }{\delta ^{2}_n} &\text{(Decreasing in $j$, take $j = 0$)} \\
&\le \frac{\sqrt{n} }{8}. &\text{(Assumption)}
\end{aligned}\] so that
\[
\frac{1}{2}\left( 2^{j - 1}\delta _n \right) ^{2} = \frac{1}{8}\left( 2^{j}\delta _n \right)^{2} \ge \frac{J\left( 2^{j}\delta _n \right)}{\sqrt{n} } + 4 \cdot 2^{j}\delta _n \sqrt{\frac{1 + t + j}{n}}.
\]
\item
We now want to consider the probability
\[\begin{aligned}
\mathbb{P} \left( \left\Vert \hat{g}_n - g ^{0} \right\Vert > \delta _n \right) &\le \mathbb{P} \left( \exists g \in \mathcal{G} , \left\Vert g - g^{0} \right\Vert _n > \delta _n \text{ s.t.} \smash[b]{\underbrace{\left\Vert g - g^{0} \right\Vert _n^{2} \le 2\hat{\varepsilon }_n ^{T} \left( g - g^{0} \right)/n}_{\text{always holds for $\hat{g}_n $} }} \right)\\[10pt]
\text{(Union + peeling)} &\le \sum_{j = 1}^{ \infty} \mathbb{P} \left( \exists g \in \mathcal{G} , 2^{j -1} \delta _n < \left\Vert g - g_0 \right\Vert \le 2^{j}\delta _n, \left\Vert g - g_0 \right\Vert _n^{2} \le 2\hat{\varepsilon }_n ^{T}\left( g - g^{0} \right)/n \right) \\
&\le \sum_{j = 1}^{\infty} \mathbb{P} \left( \sup _{g \in \mathcal{G} \left( 2^{j}\delta _n \right)}2\hat{\varepsilon }_n ^{T} \left( g - g^{0} \right)/n \ge \left( 2^{j - 1}\delta _n \right)^{2} \right)\\
\text{(Step 2.)} &\le \sum_{j = 1}^{\infty} \mathbb{P} \left( \sup _{g \in \mathcal{G} \left( 2^{j}\delta _n \right)}\hat{\varepsilon }_n ^{T} \left( g - g^{0} \right)/n \ge \frac{J\left( 2^{j}\delta _n \right)}{\sqrt{n} } + 4 \cdot 2^{j}\delta _n \sqrt{\frac{1 + t + j}{n}} \right)\\
\text{(Chaining)} &\le \sum_{j = 1}^{\infty} \exp \left[ -\left( t + j \right) \right] \\
\text{(Geom. series)} &= \frac{1}{e - 1}e^{-t} .
\end{aligned}\]
\end{enumerate}
This is Theorem 11.2.1.
[/latex]

Front: [latex]
Consider least squares linear regression. Let $\mathcal{X} = \mathbb{R} ^{r}$, and take $\mathcal{G}$ as the class of regression functions. Assume $Y_i = g^{0} (x_i) + \varepsilon _i $ with fixed $x_i$ and $\varepsilon _i$ i.i.d. $\mathcal{N} \left( 0, 1 \right)$.
Suppose you wish to put a bound in probability on the convergence rate $\left\Vert \hat{g}_n - g ^{0} \right\Vert ^{2}_n$. What properties of the space $\mathcal{G} $ do you need to consider?
[/latex]
Back: [latex]
We need to be able to bound the entropy integral
\[
J\left( \delta \right) := 2\int_{0}^{\delta } \sqrt{2H\left( u, \mathcal{G} \left( \delta \right), \left\Vert \cdot \right\Vert _n \right)} du\\
\] which relies on the entropy of a ball around $g^{0} $ not growing too fast as $u$ decreases. In other words, we want $\mathcal{G} $ to be locally compact.
This is because we want to apply Theorem 11.2.1, which states that if
\begin{enumerate}[(i)]
\item for all $\delta > 0$ the integral
\[J\left( \delta \right) := 2\int_{0}^{\delta } \sqrt{2H\left( u, \mathcal{G} \left( \delta \right), \left\Vert \cdot \right\Vert _n \right)} du\]
is finite,
\item $\frac{J\left( \delta \right)}{\delta ^{2} }$ is decreasing in $\delta $, and
\item for all $t > 0$ \[\delta _n^{2} \ge 8 \left( \frac{J\left( \delta _n \right)}{\sqrt{n} } + 4\delta _n \sqrt{\frac{1 + t}{n}} \right)\]
\end{enumerate}
then for all $t > 0$ it holds that
\[
\mathbb{P} \left( \left\Vert \hat{g}_n - g ^{0} \right\Vert_n > \delta _n \right) \le \frac{1}{e-1}\exp \left[ -t \right].
\]
[/latex]

Front: [latex]
Consider least squares linear regression. Let $\mathcal{X} = \mathbb{R} ^{r}$, $\mathcal{G} = \left\{ g(x) = x\theta : \theta \in \mathbb{R} ^{r} \right\}$. Assume $Y_i = g^{0} (x_i) + \varepsilon _i $ with fixed $x_i$ and $\varepsilon _i$ i.i.d. $\mathcal{N} \left( 0, 1 \right)$.
Argue why for the class $\mathcal{G} \left( \delta \right) := \left\{ g \in \mathcal{G} : \left\Vert g - g^{0} \right\Vert _n \le \delta \right\}$ and for some $u < \delta $ we can bound the entropy
\[
H\left( u, \mathcal{G} \left( \delta \right), \left\Vert \cdot \right\Vert _n \right) \le r\log \left( \frac{2\delta + u}{u} \right)
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item Since we are considering only linear functions, the space $\left( \mathcal{G} , \left\Vert \cdot \right\Vert _n \right)$ is Euclidean. We will start by considering the packing number $D = D\left( u, \mathcal{G} \left( \delta \right), \left\Vert \cdot \right\Vert _n \right)$. The volume of the packing balls cannot exceed the volume of the large ball with a little extra for the packing balls sticking out, which lets us put an upper bound on the packing number $D$.
\begin{center}\includegraphics[width=0.3\textwidth]{ept_linear_regression_convergence_rate}\end{center}
\item We have for some constant $C_r$
\[\begin{aligned}
D C_r \left( \frac{u}{2} \right)^{r} & \le \left( \delta + \frac{u}{2} \right)^{r}C_r\\
D &\le \left( \frac{2\delta + u }{u} \right)^{r}.
\end{aligned}\]
\item We know that for all $\delta >0$ and metric spaces $\left( S, d \right)$, the covering number is bounded by the packing number
\[
N\left( \delta , S, d \right) \le D\left( \delta , S, d \right).
\]
\item So, we can finish by writing
\[\begin{aligned}
H\left( u, \mathcal{G} \left( \delta \right) , \left\Vert \cdot \right\Vert _n \right) &= \log N\left( u, \mathcal{G} \left( \delta \right), \left\Vert \cdot \right\Vert _n \right)\\
&\le r\log \left( \frac{2\delta + u}{u} \right).
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Consider least squares linear regression. Let $\mathcal{X} = \mathbb{R} ^{r}$, $\mathcal{G} = \left\{ g(x) = x\theta : \theta \in \mathbb{R} ^{r} \right\}$. Assume $Y_i = g^{0} (x_i) + \varepsilon _i $ with fixed $x_i$ and $\varepsilon _i$ i.i.d. $\mathcal{N} \left( 0, 1 \right)$. We know that in Euclidean space, the $u$-covering entropy of a $\delta $-ball $B_\delta $ can be bounded as
\[H\left( u, B_\delta , \left\Vert \cdot \right\Vert \right) \le r\log \left( \frac{2\delta + u}{u} \right).\]
Show that for all $t > 0$,
\[
\mathbb{P} \left( \left\Vert \hat{g}_n - g ^{0} \right\Vert_n ^{2} > 8 \left( A_0 \sqrt{\frac{r}{n}} + 4\sqrt{\frac{1+t}{n}} \right) \right) \le \frac{1}{e - 1}\exp \left[ -t \right]
\] for some constant $A_0$.
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item Since we are considering linear functions, the space $\left( \mathcal{G} , \left\Vert \cdot \right\Vert _n \right)$ is Euclidean. So, we can use the given hint about entropy to bound the entropy integral
\[\begin{aligned}
J\left( \delta \right) &:= 2\int_{0}^{\delta } \sqrt{2H\left( u, \mathcal{G} \left( \delta \right), \left\Vert \cdot \right\Vert _n \right)} du\\
&\le \sqrt{r} 2 \int_{0}^{\delta } \sqrt{2} \sqrt{\log \left( \frac{2\delta + u}{u} \right)} du\\
&=2\delta \sqrt{2r} \int_{0}^{1} \sqrt{\log \left( 2 + v \right)} dv \\
&=: A_0 \sqrt{r} \delta. &\text{(For some constant $A_0$)}
\end{aligned}\]
\item Now that we have this bound, and we know $\frac{J\left( \delta \right)}{\delta ^{2}}$ is decreasing in $\delta $, we can apply Theorem 11.2.1 with
\[
\delta _n^{2} \ge 8 \left( A_0 \sqrt{\frac{r}{n}} + 4\sqrt{\frac{1 + t}{n}} \right)
\] resulting in the desired inequality for all $t > 0$
\[
\mathbb{P} \left( \left\Vert \hat{g}_n - g ^{0} \right\Vert_n^{2} > 8 \left( A_0 \sqrt{\frac{r}{n}} + 4\sqrt{\frac{1+t}{n}} \right) \right) \le \frac{1}{e - 1}\exp \left[ -t \right].
\]
\end{enumerate}
[/latex]

Front: [latex]
Consider regularized least squares regression. Let the norm be $\left\Vert v \right\Vert _n^{2} := v^{T}v/n$ for $v \in \mathbb{R} ^{n}$, the model $Y_i = g_0\left( x_i \right) + \varepsilon _i$, $i = 1, \dots, n$, with $\left\{ x_1, \dots, x_n \right\} \in \mathcal{X} ^{n}$ fixed where $g_0 : \mathcal{X} \to \mathbb{R} $ is an unknown regression function, $\varepsilon _1, \dots, \varepsilon _n$ are i.i.d. $\mathcal{N} \left( 0, 1 \right)$.
Define the regularized least squares estimator \[\hat{g}_n := \arg \min _{g \in \mathcal{G} } \left\{ \frac{1}{n} \sum_{i = 1}^{n} \left( Y_i - g\left( x_i \right) \right)^{2} + \text{pen} \left( g \right) \right\}\] where $\mathcal{G} $ is a given class of regression functions and $\text{pen} : \mathcal{G} \to \mathbb{R} _+$ is a penalty. Define also a measure of how good the fit is
\[
\tau ^{2}\left( g \right) := \left\Vert g - g^{0} \right\Vert ^{2}_n + \text{pen} \left( g \right)
\] and an oracle $g^{*}$ with
\[
\tau ^{2}\left( g^{*} \right) := \min _{g \in \mathcal{G} } \tau ^{2}\left( g \right).
\]
Show that
\[
\tau ^{2}\left( \hat{g}_n \right) \le 2\hat{\varepsilon }_n ^{T}\left( \hat{g}_n - g ^{*} \right)/n + \tau ^{2}\left( g^{*} \right).
\]
[/latex]
Back: [latex]
\begin{enumerate}[]
\item
Since $\hat{g}_n $ is the minimizer of empirical regularized loss, we get
\[\begin{aligned}
\left\Vert Y - \hat{g}_n \right\Vert ^{2}_n + \text{pen} \left( \hat{g}_n \right) &\le \left\Vert Y - g^{*} \right\Vert _n^{2} + \text{pen} \left( g^{*} \right).
\end{aligned}\]
\item
Now just expand the above
\[\begin{aligned}
&\cancel{\left\Vert \varepsilon \right\Vert _n^{2}} - 2\hat{\varepsilon }_n ^{T}\left( \hat{g}_n - \cancel{g ^{0}} \right)/n + \underbrace{\left\Vert \hat{g}_n - g^{0} \right\Vert _n^{2} + \text{pen} \left( \hat{g}_n \right)}_{\tau^{2}\left( \hat{g}_n \right)}
\\
\le& \cancel{\left\Vert \varepsilon \right\Vert _n^{2}} - 2\hat{\varepsilon }_n ^{T}\left( g^{*} - \cancel{g_0 }\right)/n + \underbrace{\left\Vert g^{*} - g^{0} \right\Vert _n + \text{pen} \left( g^{*} \right)}_{\tau^{2}\left( g^{*} \right)}\\
\end{aligned}\] and rearrange to conclude
\[
\tau ^{2}\left( \hat{g}_n \right) \le 2\hat{\varepsilon }_n ^{T}\left( \hat{g}_n - g ^{*} \right)/n + \tau ^{2}\left( g^{*} \right).
\]
\end{enumerate}
[/latex]

Front: [latex]
Consider regularized least squares regression. Let the norm be $\left\Vert v \right\Vert _n^{2} := v^{T}v/n$ for $v \in \mathbb{R} ^{n}$, the model $Y_i = g_0\left( x_i \right) + \varepsilon _i$, $i = 1, \dots, n$, with $\left\{ x_1, \dots, x_n \right\} \in \mathcal{X} ^{n}$ fixed where $g_0 : \mathcal{X} \to \mathbb{R} $ is an unknown regression function, $\varepsilon _1, \dots, \varepsilon _n$ are i.i.d. $\mathcal{N} \left( 0, 1 \right)$. Define $\tau ^{2}\left( g \right) = \left\Vert g - g^{0} \right\Vert _n^{2} + \text{pen} \left( g \right)$, with $g^{*}$ the theoretical minimizer of $\tau ^{2}$.
Suppose the entropy integrability condition
\[
J\left( \delta \right) := 2 \int_{0}^{2\delta } \sqrt{2H\left( u, \mathcal{G} \left( u \right), \left\Vert \cdot \right\Vert _n \right)} du < \infty
\] and that $J\left( \delta \right)/\delta ^{2}$ is decreasing. Let $t > 0$ and \[\delta ^{2}_n \ge 4\tau ^{2}\left( g^{*} \right) + 8 \left( \frac{J\left( \delta _n \right)}{\sqrt{n} } + 4\delta _n \sqrt{\frac{1 + t}{n}} \right).\]
Show that then we have
\[
\mathbb{P} \left( \tau \left( \hat{g}_n \right) > \delta _n \right) < \frac{1}{e - 1}\exp \left[ -t \right].
\]
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item $J\left( \delta \right)/\delta ^{2}$ decreasing implies that
\[\begin{aligned}
\left( 2^{j}\delta _n \right)^{2} &\ge 4\tau ^{2}\left( g^{*} \right) + 8 \left( \frac{J\left( 2^{j}d_n \right)}{\sqrt{n} } + 4 \cdot 2^{j}\delta _n \sqrt{\frac{1 + t + j}{n}} \right) \text{ for $j = 1, 2, \dots$}
\end{aligned}\]
\item Now look at
\[\begin{aligned}
\left( 2^{j - 1}\delta _n \right)^{2} - \tau ^{2}\left( g^{*} \right)
&= \frac{\left( 2^{j}\delta _n \right)^{2}}{4} - \tau ^{2}\left( g^{*} \right) \\
&\ge 2\left( \frac{J\left( 2^{j}\delta _n \right)}{\sqrt{n} } + 4 \cdot 2^{j}\delta _n \sqrt{\frac{1 + t + j}{n}} \right)
\end{aligned}\]
\item Now we will work to do the peeling. First, by the basic inequality, we get
\[\begin{aligned}
\mathbb{P} \left( \tau \left( \hat{g}_n \right) \ge \delta _n \right)
=& \mathbb{P} \left( \exists g : \tau ^{2}\left( g \right) \le 2\hat{\varepsilon }_n ^{T}\left( g - g^{*} \right)/n + \tau ^{2}\left( g^{*} \right) \text{ and } \tau \left( g \right) > \delta _n \right).
\end{aligned}\]
\item Take the above and do the peeling step
\[\begin{aligned}
& \mathbb{P} \left( \exists g : \tau ^{2}\left( g \right) \le 2\hat{\varepsilon }_n ^{T}\left( g - g^{*} \right)/n + \tau ^{2}\left( g^{*} \right) \text{ and } \tau \left( g \right) > \delta _n \right)\\
\le & \sum_{j = 1}^{\infty} \mathbb{P} \left( \sup _{\tau ^{2}\left( g \right) \le \left( 2^{j}\delta _n \right)^{2}} 2\hat{\varepsilon }_n ^{T}\left( g - g^{*} \right)/n \ge \left( 2^{j - 1}\delta _n \right)^{2} - \tau ^{2}\left( g^{*} \right) \right)\\
\stackrel{\text{(2.)} }{\le} & \sum_{j = 1}^{\infty} \mathbb{P} \left( \sup _{\tau ^{2}\left( g \right) \le\left( 2^{j}\delta _n \right)^{2}} \hat{\varepsilon }_n ^{T}\left( g - g^{*} \right)/n \ge \left( \frac{J\left( 2^{j}\delta _n \right)}{\sqrt{n} } + 4 \cdot 2^{j}\delta _n \sqrt{\frac{1 + t + j}{n}} \right) \right)\\
\le & \sum_{j = 1}^{\infty} \exp \left[ -\left( t + j \right) \right] = \frac{1}{e - 1}\exp \left[ -t \right]. \text{ (Chaining as in Thm. 9.2.1)}
\end{aligned}\]
\end{enumerate}
[/latex]

Front: [latex]
Consider regularized least squares regression. Let the norm be $\left\Vert v \right\Vert _n^{2} := v^{T}v/n$ for $v \in \mathbb{R} ^{n}$, the model $Y_i = g_0\left( x_i \right) + \varepsilon _i$, $i = 1, \dots, n$, with $\left\{ x_1, \dots, x_n \right\} \in \mathcal{X} ^{n}$ fixed where $g_0 : \mathcal{X} \to \mathbb{R} $ is an unknown regression function, $\varepsilon _1, \dots, \varepsilon _n$ are i.i.d. $\mathcal{N} \left( 0, 1 \right)$. Define $\tau ^{2}\left( g \right) = \left\Vert g - g^{0} \right\Vert _n^{2} + \text{pen} \left( g \right)$, with $g^{*}$ the theoretical minimizer of $\tau ^{2}$.
What are the basic steps to find a convergence rate of $\hat{g}_n $? More precisely, what do you need to do to find a probabilistic bound of form
\[
\mathbb{P} \left( \tau ^{2} \left( \hat{g}_n \right) > f(n, \cdots) \right) < h(\cdots)
\] for some functions $f, h$?
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item
We want to use the theorem
\begin{enumerate}[ ]
\item
Suppose the entropy integrability condition
\[
J\left( \delta \right) := 2 \int_{0}^{2\delta } \sqrt{2H\left( u, \mathcal{G} \left( u \right), \left\Vert \cdot \right\Vert _n \right)} du < \infty
\] and that $J\left( \delta \right)/\delta ^{2}$ is decreasing. Let $t > 0$ and \[\delta ^{2}_n \ge 4\tau ^{2}\left( g^{*} \right) + 8 \left( \frac{J\left( \delta _n \right)}{\sqrt{n} } + 4\delta _n \sqrt{\frac{1 + t}{n}} \right).\]
Then
\[\begin{aligned}
\mathbb{P} \left( \tau \left( \hat{g}_n \right) > \delta _n \right) < \frac{1}{e - 1}\exp \left[ -t \right].
\end{aligned}\]
\end{enumerate}
\item
So, we have to investigate the entropy of a ball $\mathcal{G} \left( u \right) = \left\{ g \in \mathcal{G} : \tau ^{2}\left( g \right) \le \delta ^{2} \right\}$ to see if the integral $J\left( \delta \right)$ converges. Then, once we have $J\left( \delta \right)$, we can select $\delta_n $ to fulfil
\[
\delta _n^{2} \gtrsim \frac{J\left( \delta _n \right)}{\sqrt{n} } \text{ or } \sqrt{\frac{1 + t}{n}} \cdot \delta _n \text{ or } \tau ^{2}\left( g^{*} \right).
\]
\item
Lastly, we pick some $t$, and see with probability $1 - \frac{1}{e - 1}\exp \left[ -t \right]$ the inequality
\[
\tau ^{2}\left( \hat{g}_n \right) \le \delta_n ^{2}.
\] Here, the right hand side will be the rate of convergence.
\end{enumerate}
[/latex]

Front: [latex]
Consider regularized least squares regression. Let the norm be $\left\Vert v \right\Vert _n^{2} := v^{T}v/n$ for $v \in \mathbb{R} ^{n}$, the model $Y_i = g_0\left( x_i \right) + \varepsilon _i$, $i = 1, \dots, n$, with $\left\{ x_1, \dots, x_n \right\} \in \mathcal{X} ^{n}$ fixed where $g_0 : \mathcal{X} \to \mathbb{R} $ is an unknown regression function, $\varepsilon _1, \dots, \varepsilon _n$ are i.i.d. $\mathcal{N} \left( 0, 1 \right)$. Define $\tau ^{2}\left( g \right) = \left\Vert g - g^{0} \right\Vert _n^{2} + \text{pen} \left( g \right)$, with $g^{*}$ the theoretical minimizer of $\tau ^{2}$.
Let $\mathcal{G} $ be the class
\[
\mathcal{G} := \left\{ g_\theta \left( x \right) = \sum_{j = 1}^{p} \theta _j\psi _j(x), \theta \in \mathbb{R} ^{p} \right\}
\] where $\psi _j\left( \cdot \right)$ is a known feature mapping, and the data is high dimensional $p >> n$. Define the penalty as the number of nonzero parameters
\[
\text{pen} \left( g_\theta \right) := \lambda ^{2} \#\left\{ \theta _j \neq 0 \right\}.
\]
Find the optimal $\lambda $ and determine its convergence rate, as in find the $K_n$ to bound $\tau ^{2}\left( \hat{g}_n \right) \le K_n$ with probability $1 - \frac{1}{e -1} \exp \left[ -t \right]$ in terms of parameter $t$, dimensionality $p$, sample size $n$, and target function $g^{0} $. You may dismiss the constants.
[/latex]
Back: [latex]
\begin{enumerate}[1.]
\item
We will consider the entropy of the local class $\mathcal{G} \left( \delta \right) = \left\{ g \in \mathcal{G} : \tau ^{2}\left( g \right) \le \delta ^{2} \right\}$ for $\delta > 0$. In this ball, we have \[\tau ^{2}\left( g_\theta \right) = \left\Vert g_\theta - g^{0} \right\Vert ^{2} + \lambda ^{2}\#\left\{ \theta _j \neq 0 \right\} \le \delta ^{2}.\] This implies
\begin{enumerate}[(i)]
\item $\left\Vert g_\theta - g^{*} \right\Vert _n \le \delta$ since $g^{*}$ is the minimizer/center of the ball, and
\item $\#\left\{ \theta _j \neq 0 \right\} \le \frac{\delta ^{2}}{\lambda ^{2}} =: r$. For simplicity we will take $r$ as the smallest integer which satisfies the inequality.
\end{enumerate}
\item So the entropy of this class can be bounded
\[\begin{aligned}
H\left( u, \mathcal{G} \left( \delta \right), \left\Vert \cdot \right\Vert _n \right) &\le \underbrace{r \log \left( \frac{2\delta + u}{u} \right)}_{\substack{\text{ball packing argument} \\ \text{based on volume}} } + \underbrace{r\log p}_{\substack{\text{choices of nonzero $\theta _j$} \\ \log \binom{p}{r} \le r\log p} }.
\end{aligned}\]
\item Now we can proceed to the entropy integral
\[\begin{aligned}
J\left( \delta \right) &:= 2\sqrt{2} r \int_{0}^{2\delta } \sqrt{\log \left( \frac{2\delta + u}{u} \right)} du + 2\sqrt{2} \sqrt{r} \sqrt{\log p} \cdot 2\delta \\
&=2\sqrt{2} \sqrt{r} \cdot 2\delta \left[ \int_{0}^{1} \sqrt{\log \left( 1 + \frac{1}{u} \right)} du + \sqrt{\log p} \right].
\end{aligned}\]
\item Since $r = \frac{\delta ^{2}}{\lambda ^{2}}$, we can write
\[\begin{aligned}
J\left( \delta \right) &= A_0\frac{\delta ^{2}}{\lambda} \sqrt{\log p}
\end{aligned}\]with $p \ge e$ and some constant $A_0$.
\item We want to now find $\delta _n$ for the convergence rate bound. We will base it on the condition
\[
\delta _n^{2} \ge 4 \tau ^{2}\left( g^{*} \right) + 8 \left( \frac{J\left( \delta \right)}{\sqrt{n} } + 4 \cdot \delta _n \sqrt{\frac{1+t}{n}} \right)
\] which we know then gives a bound of
\[
\mathbb{P} \left( \tau \left( \hat{g}_n \right) > \delta _n \right) < \frac{1}{e-1}\exp \left[ -t \right].
\]
So, dismissing the constants, we have
\[
\delta _n^{2} \gtrsim \underbrace{ \frac{J\left( \delta _n \right)}{\sqrt{n} }}_{\left( \dagger.1 \right)} \text{ or } \underbrace{\sqrt{\frac{1 + t}{n}} \cdot \delta _n}_{\left( \dagger.2 \right)} \text{ or } \tau ^{2}\left( g^{*} \right).
\]
\item We can use ($\dagger$.1) to condition the tuning parameter $\lambda $
\[
\delta ^{2}_n \gtrsim \frac{\delta _n^{2}}{\lambda }\sqrt{\frac{\log p}{n}} \implies \lambda \gtrsim \sqrt{\frac{\log p}{n}} .
\] So, we set the optimal $\lambda = A_\lambda \sqrt{\frac{\log p}{n}} $ for some constant $A_\lambda $.
\item ($\dagger$.2) then lets us condition $\delta _n$
\[
\delta _n^{2} \ge \sqrt{\frac{1 + t}{n}} \delta _n \implies \delta _n \ge \sqrt{\frac{1 + t}{n}} .
\]
\item Now we get the desired convergence rate inequality. With probability $1 - \frac{1}{e - 1}\exp \left[ -t \right]$ and for some constant $A_1$, we can say
\[\begin{aligned}
\tau ^{2} \left( \hat{g}_n \right) &\le \left( \tau ^{2}\left( g^{*} \right) + \frac{1+t}{n} \right) A_1 \\
&\lesssim \left\Vert g^{*} - g_0 \right\Vert ^{2}_n + \lambda ^{2} \#\left\{ \theta _j^{*} \neq 0 \right\} + \frac{1 + t}{n}\\
&\le \lambda ^{2} \#\left\{ \theta _j^{0} \neq 0 \right\} + \frac{1 + t}{n} &\text{($g^{*}$ is minimizer, so set to $g^{0} $)} \\
&\eqsim \frac{\log p}{n} \#\left\{ \theta _j^{0} \neq 0 \right\} + \frac{1 + t}{n} .
\end{aligned}\]
\end{enumerate}
[/latex]