Hilbert space
The algebra of observables in quantum mechanics is naturally an algebra of operators defined on a Hilbert space, according to Werner Heisenberg's matrix mechanics formulation of quantum theory. Von Neumann began investigating operator algebras in the 1930s, as rings of operators on a Hilbert space. The kind of algebras studied by von Neumann and his contemporaries are now known as von Neumann algebras. In the 1940s, Israel Gelfand, Mark Naimark and Irving Segal gave a definition of a kind of operator algebras called C*-algebras that on the one hand made no reference to an underlying Hilbert space, and on the other extrapolated many of the useful features of the operator algebras that had previously been studied. The spectral theorem for self-adjoint operators in particular that underlies much of the existing Hilbert space theory was generalized to C*-algebras. These techniques are now basic in abstract harmonic analysis and representation theory.
Observable
If the Hilbert space is finite-dimension, it can be represented by a Hermitian matrix if the space is finite-dimensional. In an infinite-dimensional Hilbert space, the observable is represented by a symmetric operator, which may not be defined everywhere. The reason for such a change is that in an infinite-dimensional Hilbert space, the observable operator can become unbounded, which means that it no longer has a largest eigenvalue. This is not the case in a finite-dimensional Hilbert space: an operator can have no more eigenvalues than the dimension of the state it acts upon, and by the well-ordering property, any finite set of real numbers has a largest element. For example, the position of a point particle moving along a line can take any real number as its value, and the set of real numbers is uncountably infinite. Since the eigenvalue of an observable represents a possible physical quantity that its corresponding dynamical variable can take, we must conclude that there is no largest eigenvalue for the position observable in this uncountably infinite-dimensional Hilbert space. 
A crucial difference between classical quantities and quantum mechanical observables is that the latter may not be simultaneously measurable, a property referred to as complementarity. This is mathematically expressed by non-commutativity of the corresponding operators, to the effect that the commutator