Mathematics

The Tensor Product: from vector spaces to categories

  1. Introduction
  2. General Definition
  3. The tensor product is not the same as the Cartesian product
    1. A jump to categories
  4. Applications
    1. TensorFlow
    2. Artificial Intelligence
    3. Quantum Optics

Introduction

For any vectors $\left\langle x_{1},x_{2}\right\rangle $ of $\mathbb{R}^{2}$ and $\left\langle y_{1},y_{2},y_{3}\right\rangle $ of $\mathbb{R}^{3}$, a product of these two vectors, which is denoted as $\left\langle x_{1},x_{2}\right\rangle \otimes\left\langle y_{1},y_{2},y_{3}\right\rangle $, is defined as the matrix
\[
\left[\begin{array}{ccc}
x_{1}y_{1} & x_{1}y_{2} & x_{1}y_{_{3}}\\
x_{2}y_{1} & x_{2}y_{2} & x_{2}y_{3}
\end{array}\right].
\]
The collection of all the products $\left\langle x_{1},x_{2}\right\rangle \otimes\left\langle y_{1},y_{2},y_{3}\right\rangle $, up to some equivalence, is called the tensor product of $\mathbb{R}^{2}$ and $\mathbb{R}^{3}$ and is denoted by $\mathbb{R}^{2}\otimes\mathbb{R}^{3}$.

General Definition

For any vector spaces $U,V,W$ over a field $\mathbb{F}$ and any bilinear map $f$ from the Cartesian product $U\times V$ to $W$, and for any basis $B_{U}$ of $U$ and basis $B_{V}$ of $V$, we have this result, which can be checked by expanding the definitions of a basis and of a bilinear map:

Theorem 1: If $f$ is onto, the collection of all $f\left(x,y\right)$, with $x$ in $B_{U}$ and $y$ in $B_{V}$, is a basis of $W$.

Here is a definition of the tensor product: the vector space $W$ with the basis consisting of all $f\left(x,y\right)$, with $x$ in $B_{U}$ and $y$ in $B_{V}$, is called the tensor product of $U$ and $V$ whenever $f$ is onto, and $W$ is denoted in terms $U$ and $V$ as $U\otimes V$.

The tensor product is not the same as the Cartesian product

The Cartesian product $U\times V$ of any two vector spaces $U,V$ over the same field is a vector space over that field with addition defined as $\left\langle x_{1},y_{1}\right\rangle +\left\langle x_{2},y_{2}\right\rangle :=\left\langle x_{1}+x_{2},y_{1}+y_{2}\right\rangle $ and scalar multiplication defined as $k\left\langle x_{1},y_{1}\right\rangle :=\left\langle kx_{1},ky_{1}\right\rangle $.

We will show you that the Cartesian product is not the same as the tensor product: we will exhibit two vector spaces whose Cartesian product is not isomorphic to their tensor product.

We take the vector spaces $U,V$ to be the field $\mathbb{F}_{2}$ over itself. The field $\mathbb{F}_{2}$ has two elements $0$ and $1$ with multiplication and addition defined as
\[
\begin{array}{cccccccc}
\times & 0 & 1 & & & + & 0 & 1\\
0 & 0 & 0 & & & 0 & 0 & 1\\
1 & 0 & 1 & & & 1 & 1 & 0
\end{array}.
\]

The addition and scalar multiplication on $U$ and $V$ are exactly as defined above, and the basis of $\mathbb{F}_{2}$ is the singleton $\left\{ 1\right\} $. The elements of the Cartesian product $\mathbb{F}_{2}\times\mathbb{F}_{2}$ are the collection of pairs $\left\{ \left\langle 0,0\right\rangle ,\left\langle 0,1\right\rangle ,\left\langle 1,0\right\rangle ,\left\langle 1,1\right\rangle \right\} $.

The subcollection $\left\{ \left\langle 0,1\right\rangle ,\left\langle 1,0\right\rangle \right\} $ spans $\mathbb{F}_{2}\times\mathbb{F}_{2}$ and is linearly independent, so it is a basis. For the tensor product $\mathbb{F}_{2}\otimes\mathbb{F}_{2}$, recall from the above definition that there is an onto bilinear map $f$ from $\mathbb{F}_{2}\times\mathbb{F}_{2}$ to $\mathbb{F}_{2}\otimes\mathbb{F}_{2}$, so by Theorem 1, the singleton $\left\{ f\left(1,1\right)\right\} $ is a basis of the tensor product. Since $\mathbb{F}_{2}\times\mathbb{F}_{2}$ has a basis with $2$ elements and $\mathbb{F}_{2}\otimes\mathbb{F}_{2}$ has a basis with $1$ element, they cannot be isomorphic.

A jump to categories

The Cartesian product and the tensor product can be seen as bifunctors from $\mathbb{F}- \mathbf{Vect}\times\mathbb{F}-\mathbf{Vect}$ to $\mathbb{F}-\mathbf{Vect}$, where $\mathbb{F}-\mathbf{Vect}$ is the category of vector spaces over a common field $\mathbb{F}$. Since we have shown above that $\mathbb{F}_{2}\times\mathbb{F}_{2}$ is not isomorphic to $\mathbb{F}_{2}\otimes\mathbb{F}_{2}$, we have the following:

Corollary 2: As bifunctors, the Cartesian product and the tensor product are not naturally isomorphic.

Remark: The tensor product can be defined for other structures, such as modules, topological spaces, graphs, and matrices.

Applications

TensorFlow

Tensors are elements of a tensor product of vector spaces. TensorFlow  is a software library, which performs computations on tensors. TensorFlow is used in machine learning applications, such as building neural networks.

Artificial Intelligence

In this paper on natural language processing, the tensor product is used to represent strings of words in a vector space.

Quantum Optics

The displacement operator   on several modes can be decomposed as a tensor product. For instance, see this paper, section III.

———————————————————————————————————————————————————————–



Leave a Reply

Your email address will not be published. Required fields are marked *