Firstly, let’s make a little speculations, using our intuition. We, sure, if $\sum\limits_{i = 1}^{n}{x_i} = n$, that $H_0$ is false, because we haven’t reason for assume, that coin can be $0$, therefore $p = 1$. But, what about case, when we have $n = 20$ and $\sum{x_i} = 15$? In this case, we can’t be sure. Our questions, it’s, how to make a decision in such situations. In this decision we may make a one of two types of errors:

And in statistics our default predisposition it’s minimization errors of type 1. For example, if we made a medicine drug and after testing we checking $H_0$: “Our medicine drug is effective.”. If we make error of type 1, we will put broken drug to many peoples — it’s not good. In business, if make first type error we need to spend more many for new model. That’s the reason, for:

For make this decision we need to compare $\sum{x_i}$ with $0.5n$. How big is difference? The theorem of Moivre–Laplace:

Let’s use it in our case:

For example, if in real experiment we got $Z = 10$, that’s mean $H_0$ is very unlikely. So, we need to define the border $t$ for $Z$, after that we reject $H_0$: We need to find the value of $t$, that probability of cross over it, is significance level $\alpha$. If $|Z| > q_{1 - \frac{\alpha}{2}}$, we talk $H_1$ is truth and we reject $H_0$. Else, we approve $H_0$.

The interesting fact is first type of errors very easy for control. We can always say, that $H_0$ is truth. Then probability of errors type 1 will equal zero.

The problem settings is pretty simple: we want to calculate statistics (mean, variance, covariance, correlation) step by step, when we receive $n$ data in a stream, without saving all data and additional iteration. Let’s remind formulas on base way. Mean:

Variance:

As you see for mean we need sum: first problem — how to get sum without all numbers? And for variance we need mean: second problem — how to know mean without all data?

# Intuition of method

Let’s try to feel the intuition of Welford method. Imagine we have the dataset: $A = \{1, 2, 3\}$, mean of this dataset is $\overline{x}_A = \frac{1 + 2 +3}{3} = 2$ and then we got one more number: $6$. Do you feel it will be move our mean to the bigger side? But how much bigger? Imagine similar dataset but with more data: $B = \{1, 1, 2, 2, 3, 3\}$ the mean also is $2$ and we also adding number $6$. Again we fill it will be move to the bigger side, but no so strong. Let’s check our intuition:

Our intuition was right! It remains for us to figure out how we can calculate this delta, if we got only previous mean $\overline{x}_{i - 1}$, new number $x_i$ and count of our numbers. The Welford method told: every new value move mean in its own direction with weight $1/i$:

Let’s check it for $B$ dataset. We got mean without new number $\overline{x}_{i - 1} = 2$ and new number $x_i = 6$ and it’s a seventh number, $i = 7$. Goal is find a $\overline{x}_i$ mean with Welfod forumula:

It’s absolutely exact value, as you see. Nice!

# Variance, correlation, covariance

The real power of Welford’s algorithm reveals itself when calculating variance. The naive approach requires storing all data points and recalculating the mean before computing variance, which is computationally expensive. Welford’s method elegantly solves this by maintaining a running estimate of variance using the following recurrence relation:

For variance, we maintain the current mean $\overline{x}$ and the sum of squares of differences $S$. For each new data point $x_n$ at step $n$:

The population variance can then be calculated as $\sigma^2 = S_n/n$, while the sample variance is $s^2 = S_n/(n-1)$.

This approach extends beautifully to covariance and correlation. For two variables $X$ and $Y$, we maintain counts $n$, means $\overline{x}$ and $\overline{y}$, and accumulated products $C_{xy}$:

The covariance is then $\text{cov}(X,Y) = \frac{C_{xy_n}}{(n-1)}$ and correlation is $\text{corr}(X,Y) = \frac{C_{xy_n}}{\sqrt{S_{x_n} S_{y_n}}}$.

# C implementation

Here’s a complete C implementation of Welford’s algorithm for calculating mean and variance:

#include <stdio.h>
#include <math.h>

typedef struct {
    long long n;
    double mean;
    double M2;
} welford_state;

void welford_update(welford_state *state, double x) {
    state->n++;
    double delta = x - state->mean;
    state->mean += delta / state->n;
    state->M2 += delta * (x - state->mean);
}

double welford_mean(const welford_state *state) {
    return state->mean;
}

double welford_variance(const welford_state *state) {
    return (state->n > 1) ? state->M2 / (state->n - 1) : 0.0;
}

double welford_stddev(const welford_state *state) {
    return sqrt(welford_variance(state));
}

int main() {
    welford_state state = {0, 0.0, 0.0};
    double test_data[] = {1.0, 2.0, 3.0, 6.0};
    size_t data_size = sizeof(test_data) / sizeof(test_data[0]);
    
    for (size_t i = 0; i < data_size; i++) {
        welford_update(&state, test_data[i]);
        printf("Added %.1f: mean=%.3f, variance=%.3f, stddev=%.3f\n",
               test_data[i], welford_mean(&state), 
               welford_variance(&state), welford_stddev(&state));
    }
    
    return 0;
}

For covariance and correlation, we can extend this approach:

typedef struct {
    long long n;
    double mean_x, mean_y;
    double C2;
} welford_cov_state;

void welford_cov_update(welford_cov_state *state, double x, double y) {
    state->n++;
    double delta_x = x - state->mean_x;
    double delta_y = y - state->mean_y;
    state->mean_x += delta_x / state->n;
    state->mean_y += delta_y / state->n;
    state->C2 += delta_x * (y - state->mean_y);
}

double welford_covariance(const welford_cov_state *state) {
    return (state->n > 1) ? state->C2 / (state->n - 1) : 0.0;
}

We have source vectors $v_1, v_2, \dots, v_n$, and we need to get orthogonal vectors $u_1, u_2, \dots, u_n$. In general, describe the method: 0. Remove all zero vector.

1. We take first vector unchanged: $u_1 = v_1$.
2. Next vector find from $v_2$ and projections to last vectors: $u_2 = v_2 - \text{Proj}_{u_1}{v_2}$.
3. Again: $u_3 = v_3 - \text{Proj}_{u_1}{v_3} - \text{Proj}_{u_2}{v_3}$.
4. And continue it to $u_n$ vector: $u_n = v_n - \text{Proj}_{u_1}{v_n} - \text{Proj}_{u_2}{v_n} - \dots - \text{Proj}_{u_{n - 1}}{v_n}$. Let’s remind how to find projection: $\text{Proj}_{u}{v} = \frac{\langle u, v\rangle}{||u||^2}u$. If we want orthonormal basis we can product all vectors $u$ to $\frac{1}{||u_i||}$.

Let’s see on example: $v_1 = \begin{pmatrix}1 \\ 1\end{pmatrix}, \ v_2 = \begin{pmatrix}1 \\ 0\end{pmatrix}$. Firstly set $u_1 = v_1$, then $u_2 = v_2 - \text{Proj}_{u_1}{v_2} = \begin{pmatrix}1 \\ 0\end{pmatrix} - \begin{pmatrix}0.5 \\ 0.5\end{pmatrix} = \begin{pmatrix}0.5 \\ -0.5\end{pmatrix}$. Finally we got orthogonal vectors $u_1 = \begin{pmatrix}1 \\ 0\end{pmatrix}$, $u_2 = \begin{pmatrix}0.5 \\ -0.5\end{pmatrix}$.

See on example: $f(x, y, z) = x^2 + y^2 - 4x - 4y + z^4 - 4z^2$. Let’s find gradient:

Then set equal to zero: $\nabla f = 0$, and got system of equations:

Finally, we got this stationary points: $(2, 2, 0), \ (2, 2, \sqrt{2}), \ (2, 2, -\sqrt{2})$.

If you want to continue analysis, use Hessian matrix.

Now, look at case, when we have function and constraints. This is where the Lagrange multiplier method comes into play. For example, function is $f(x, y) = x^2 + y^2$ on condition $x + y = 1$. Rewrite a condition as: $g(x, y) = 0$, therefore $g(x, y) = x + y - 1$. Then, find gradients for all functions:

After that, write down the Lagrange equation and solve it. In general, Lagrange equation looks like:

In our case:

Done! Stationary point is $\left(\frac{1}{2}, \frac{1}{2} \right)$. Let’s resume: if functions without constraints use equation $\nabla f = 0$, else use Lagrange multiplier method.

How to find stationary points of the function you can read at this article: “Stationary points and Lagrange method”

Let’s learn it on example. Function and stationary points:

Firstly, build Hessian matrix. In general form Hessian matrix looks like:

For our function it is: $H = \begin{bmatrix}6x & -3 \\-3 & 6y\end{bmatrix}$. Then substitute points in this matrix and find eigenvalues:

Then use this rules:

• If Hessian eigenvalues is positives, point is local minimum
• If Hessian eigenvalues is negative, point is local maximum
• If Hessian eigenvalues not defined, point is saddle

As result, point $A$ it is saddle point and point $B$ it is local minimum.

Let’s consider the plane $\pi$ in $\mathbb{R}^3$ space, which passes through zero-coordinate. Consider two non-collinear vectors $u$ an $v$ from this plane. Then, any point $x \in \pi$ we can be represented as: $x = \alpha u + \beta v$.

Okay, it’s mean we can define plane $\pi$ as:

Let’s generalize the concept of a plane passing through the origin to a space of arbitrary dimension: The linear Span of vectors $v_1, v_2, \dots, v_k \ \in \ \mathbb{R}^n$ is called set of vectors, represented in form of sum of vectors $v_1, v_2, \dots, v_k$ with arbitrary ratio:

Expression $\alpha_1 v_1 + \alpha_2 v_2 + \dots + \alpha_k v_k$ will call linear combination of vectors $v_1, v_2, \dots, v_k$.

Let’s final this with pivotal definition: linear subspace $\mathbb{R}^n$ called span of finite numbers of vectors $v_1, v_2, \dots v_k \ \in \mathbb{R}^n$:

The span concept and subspace concept equivalents. And any span and subspace contain a $0$, because we can make all ratio $\alpha_1, \alpha_2, \dots, \alpha_k$ equal to zero .

Why linear subspace is important? Here is an example:

Let us take a black-and-white digital photograph of size 512 by 512 pixels. In computer memory, this photograph can be represented as $512 \times 512 = 262144$ numbers — each representing the intensity of the corresponding pixel, ranging from 0 (completely black) to 1 (completely white). Thus, the photograph can be represented as a vector $x$ in a space of dimension $d=262144$.

Now, let us consider an important practical problem — compressing the photograph without significant loss. It turns out that this problem can be solved by finding a special low-dimensional subspace and projecting the vector $x$ onto this subspace.

We told, that subspace need to contain zero. It’s mean line $y = 2x$ is a subspace, but line $y = x + 2$ not. But many properties of subspace saved when it doesn’t contain zero. Therefore, we call such subspaces as affine subspaces. Affine subspaces of $\mathbb{R}^n$ is a set of vectors $A$ of the form $a + v$, where $a \in \mathbb{R}^n$ — is a fixed vector, and vector $v$ belongs to linear subspace $V$. Let linear subspace $V$ is span of vectors $v_1, v_2, \dots, v_k$. Then, affine affine subspace is a set:

Let $V$ — subspace in $\mathbb{R}^n$. Dimension of this subspace is minimal count of vectors $v_1, v_2, \dots, v_k \in \mathbb{R}^n$ for define subspace $V$:

Dimension is denoted by: $\dim{(V)}$. Let’s think about it. For example, we have two non-collinear vectors $u, v \in \mathbb{R}^{100}$ and subspace $V = \text{span}(u, v)$. We know every vector $x \in V$ have 100 coordinates, but, by subspace definition: $x = \alpha u + \beta v$. it’s mean any vector $x$ defined only two parameters $\alpha$ and $\beta$, therefore $\dim{(V))} = 2$. For define point in $\mathbb{R}^n$ we need $n$ parameters, for define point in $V$ we need $\dim{(V)}$ parameters.

Exist subspace, which defined only one vector $0$: $V = \text{span}(0)$. We call this subspace trivial, and considered $\dim{(V)} = 0$.

If we have subspace $V = \text{span}(v_1, v_2, \dots, v_k)$, when $\dim{(V)}$ can be smaller than $k$? if and only if one of the vectors can be express as linear combination rest vectors, for example: $v_1 = \gamma_2v_2 + \gamma_3v_3 + \dots + \gamma_kv_k$.

Set of vectors called linearly independent if equality:

holds only for $\alpha_1 = \alpha_2 = \dots = \alpha_k = 0$. If equality can be holds for non-zero rates, vectors linearly dependent.

Consider space $\mathbb{R}^n$ and a given non-zero vector $w \in \mathbb{R}^n$ . Then set of vectors $P$, perpendicular $w$:

is linear subspace with $\dim{(P)} = n - 1$

# The line in plane and space

How to define a line in plane? Of course, through two points. But it’s not all. Exists are three more methods:

1. Parametric equation Let line $l$ set by two points $r_0 = (x_0; y_0)$ and $r_1 = (x_1;y_1)$. Observe that vector $a = r_0 - r_1$ in same direction with $l$. Therefore, any point on $l$ may define as $r = r_0 + ta$, where $t \in \mathbb{R}$ and full line $l$ way be with $-\infty < t < \infty$. This expression we call parametric equation:

   $$ l = \{r: r = r_o + at, \ t \in \mathbb{R} \} $$

2. Linear equation Let’s write the parametric equation using coordinates:

   $$ \begin{bmatrix}x \\ y\end{bmatrix} = \begin{bmatrix}x_0\\y_0\end{bmatrix} + \begin{bmatrix}t \cdot a_1 \\ t \cdot a_2\end{bmatrix} \Rightarrow \begin{cases}x = x_0 + ta_1 \\ y = y_0 + ta_2\end{cases} $$

   Suppose that, $a_1 \neq 0$ and $a_2 \neq 0$ and solve $t$ from each equation: $\frac{x - x_0}{a_1} = \frac{y - y_0}{a_2} \Rightarrow a_2x - a_1y + (a_1y_0 - a_2x_0) = 0$. Since $a_1, a_2, x_0, x_1$ it’s a numbers, that mean we can define it as linear equation:

   $$ Ax + By + C = 0 $$

   where $A = a_2$, $B = -a_1$, $C = (a_1y_0 - a_2x_0)$

3. Normal and translation Last method. Let’s select random point on line $l$ with radius-vector $r_0$. Let’s draw a vector $n$ from point $r_0$, that is perpendicular line $l$. Then for any point $r$ on line $l$ vectors $r - r_0$ and $n$ will be perpendicular. Rewrite it with scalar product:

   $$ l = \{ r: \langle r - r_0;n\rangle = 0\} $$

   Of course, $r_0$ it can by any point on line $l$ and normal vector $n$ any vector perpendicular to line $l$. To find normal to line, we need direction vector $a = \begin{pmatrix}a_1 \\ a_2\end{pmatrix}$ from parametric equation and then normal vector will be: $n = \begin{bmatrix}-a_2\\ a_1\end{bmatrix}$ or $n’ = -n = \begin{bmatrix} a_2 \\ -a_1\end{bmatrix}$

Let’s resume and see the relationship between this methods. If line passes through two points $r_0 = (x_0; y_0)$ and $r_1 = (x_1; y_1)$, then his can be define of of the three methods:

• parametric equation : $$ r = r_0 + ta = \begin{bmatrix}x_0\\y_0\end{bmatrix} + t \cdot \begin{bmatrix}x_1 - x_0 \\ y_1 - y_0\end{bmatrix} = \begin{bmatrix}x_0\\y_0\end{bmatrix} + \begin{bmatrix}t \cdot a_1 \\ t \cdot a_2\end{bmatrix} $$
• linear equation: $$ Ax + By + C = \underbrace{a_2}_{A} x + \underbrace{(-a_1)}_{B} y + \underbrace{(a_1 y_0 - a_2 x_0)}_{C} = 0; $$
• normal and translation: $$ \langle r - r_0; n\rangle = \left\langle \begin{bmatrix}x - x_0\\y - y_0\end{bmatrix}; \begin{bmatrix}a_1 \\ -a_1\end{bmatrix}\right\rangle = \left\langle\begin{bmatrix}x - x_0 \\ y - y_0\end{bmatrix}; \begin{bmatrix}A \\ B\end{bmatrix} \right\rangle $$ What about line in space? Parametric equation doesn’t change, one vectors will get longer. Linear equation will change: now it’s not one equation, it’s system of two equations: $$ \begin{cases} a_2x - a_1y + (a_1y_0 - a_2x_0) = 0 \\ a_3y - a_2z + (a_2z_0 - a_3y_0) = 0 \end{cases} $$ Method with normal and translation not working in space!

Addition: let’s take to look at distance from point to line:

And we can detect where point relative to a line: group of points from one side, for which $Ax +By + C > 0$ is true; group from another side, for which $Ax +By + C < 0$ is true.

# The plane in space

Again this question. How to define a plane in space? Of course, through with three points non-collinear, that do not lie on the same straight line. Collinear points lie on the same line. And again let’s consider three methods:

1. Parametric equation: $$ \pi = \{r: r = r_0 + t_1v_1 + t_2v_2, \quad t_1, t_2 \in \mathbb{R}\} $$
2. Normal and translation is the same as for line: $\langle r - r_0; n\rangle = 0$
3. Linear equation: start from normal and translation and expand it: $$ \langle r - r_0; n\rangle = 0 \quad \Rightarrow \quad \underbrace{n_1}_{A} x\ + \ \underbrace{n_2}_{B} y\ + \ \underbrace{n_3}_{C} z \ \underbrace{- \langle r_0 ;n\rangle}_{D} = 0. \ \Rightarrow \ Ax + By + Cz + D = 0 $$ Let’s look to relationship between this methods:

• For rewrite parametric equation to normal and translation we need to find normal vector from this system of equations: $\begin{cases}\langle n; v_1\rangle = 0 \\ \langle n; v_2\rangle = 0\end{cases}$
• For rewrite normal and translation form to linear equation we can use this: $A = n_1$, $B = n_2$, $C = n_3$, $D = -\langle n ;r_0\rangle$
• If we have three points, defined plane, we can go to parametric equation: $\begin{array}a v_1 = r_1 - r_0\\ v_2 = r_2 - r_0\end{array}$

Projection point $P$ on plan $\pi$ we call point $P_\pi \in \pi$, the closest to the point $P$: $P_\pi = \arg{\min{||P - X||}}, \ X \in \pi$. Then, the distance from point $P$ to plane $\pi$ will be $d(P, \pi)$ — the distance between point $P$ and her projection $P_\pi$:

Detecting where point relative a plane is the same as line.

# The basic operations of vectors

1. Sum of vectors is a sum of corresponding coordinates: $$ v + w = \begin{pmatrix}v_1 \\ v_2 \\ \ldots \\ v_n \end{pmatrix} + \begin{pmatrix}w_1 \\ w_2 \\ \ldots \\ w_n \end{pmatrix} = \begin{pmatrix}v_1 + w_1 \\ v_2 + w_2 \\ \ldots \\ v_n + w_n \end{pmatrix} $$

We can add vectors only they have same dimension. 2. Product of vector by scalar is a product off each coordinates by a scalar:

3. Difference of vectors is a sum of vectors, but one of vector product by scalar $-1$. 4. Scalar product of vectors:

Properties of scalar vectors product:

• Commutativity: $\langle v; w\rangle = \langle w; v\rangle$
• Distributivity over addition: $\langle x; (v + w)\rangle = \langle x; v\rangle + \langle x; w \rangle$
• Compatibility with scalar multiplication: $\langle v; \lambda w\rangle = \lambda \langle v;w\rangle$
• If scalar product equal zero we call this two vectors orthogonal or perpendicular

# Geometric meaning

In geometry vector is a segment having a direction.

The length of vector we calculate with Pythagoras’s theorem: $||v|| = \sqrt{\sum\limits_{i = 1}^{n}{x_i}}$. If length of vector equal one we call it a unit vector. If $v$ not a unit vector we can change it multiplying by $\lambda = \frac{1}{||v||}$. We can calculate distance and angle between two vectors. Distance is a length of difference two vectors: $d = ||v - w|| = \sqrt{\sum\limits_{i = 1}^{n}{(v_i - w_i)^2}}$. Angle between two vectors follows from Law of Cosines:

Often be helpful: $||u|| = \sqrt{\langle u;u\rangle}$. Next operation is projection vector $u$ to vector $w$:

Another property of vectors: module of scalar product does not exceed product of their length — the Cauchy–Schwarz inequality:

The Triangle inequality: $||v + w|| \leqslant ||v|| + ||w||$, so length of sum of vectors less than or equal to sum of vectors lengths. If vectors is orthogonal, work the Pythagoras’s theorem: $||a + b||^2 = ||a||^2 + ||b||^2$.