The cardiac extracellular matrix (ECM) is the three-dimensional scaffold that defines the geometry and muscular architecture of the cardiac chambers and transmits forces produced during the cardiac cycle throughout the heart wall. The cardiac ECM is an active system that responds to the stresses to which it is exposed and in the normal heart is adapted to facilitate efficient mechanical function. There are marked differences in the short- and medium-term changes in ventricular geometry and cardiac ECM that occur as a result of volume overload, hypertension, and ischemic cardiomyopathy. Despite this, there is a widespread view that a common remodeling “phenotype” governs the final progression to end-stage heart failure in different forms of heart disease. In this review article, we make the case that this interpretation is not consistent with the clinical and experimental data on the topic. We argue that there is a need for new theoretical and experimental models that will enable stresses acting on the ECM and resultant deformations to be estimated more accurately and provide better spatial resolution of local signaling mechanisms that are activated as a result. These developments are necessary to link the effects of structural remodeling with altered cardiac mechanical function.