Generalizing Atiyah extensions, we introduce and study differential abelian tensor categories over differential rings. By a differential ring, we mean a commutative ring with an action of a Lie ring by derivations. In particular, these derivations act on a differential category. A differential Tannakian theory is developed. The main application is to the Galois theory of linear differential equations with parameters. Namely, we show the existence of a parameterized Picard–Vessiot extension and, therefore, the Galois correspondence for many differential fields with, possibly, non-differentially closed fields of constants, that is, fields of functions of parameters. Other applications include a substantially simplified test for a system of linear differential equations with parameters to be isomonodromic, which will appear in a separate paper. This application is based on differential categories developed in the present paper, and not just differential algebraic groups and their representations.