More generally, a matrix group is any subgroup of the general linear group. A tensor representation of a matrix group is any representation that is contained in a tensor representation of the general linear group. For example, the orthogonal group O(n) admits a tensor representation on the space of all trace-free symmetric tensors of order two. For orthogonal groups, the tensor representations are contrasted with the spin representations.
References
Roe Goodman; Nolan Wallach (2009), Symmetry, representations, and invariants, Springer, chapters 9 and 10.
Bargmann, V., & Todorov, I. T. (1977). Spaces of analytic functions on a complex cone as carriers for the symmetric tensor representations of SO(n). Journal of Mathematical Physics, 18(6), 1141–1148.