Patients with Anorexia Nervosa (AN) and athletes share intense physical activity and pituitary hormonal disturbances related to absolute (AN) or relative (athletes) undernutrition. Pituitary gland (PG) structure evaluations in those conditions are scarce, and did not differentiate anterior from posterior lobe. We evaluated the structure-function relationship of anterior and posterior PG in AN and athletes, and potential reversibility of this alteration in a group of weight-recovered patients (AN_Rec). Manual delineation of anterior (AP) and posterior (PP) PG was performed on T1-weighted MR images in 17 women with AN, 15 women with AN_Rec, 18 athletes women and 25 female controls. Anthropometric, hormonal, and psychometric parameters were explored and correlated with PG volumes. AP volume (APV) was lower in AN (448 ± 82 mm3), AN_Rec (505 ± 59 mm3), and athletes (540 ± 101 mm3) vs. Controls (615 ± 61 mm3, p < 0.00001, p < 0.00001 and p = 0.02, respectively); and smaller in AN vs. AN_Rec (p = 0.007). PP volume did not show any differences between the groups. APV was positively correlated with weight (R = 0.36, p = 0.011) in AN, and luteinizing hormone (R = 0.35, p = 0.014) in total group. In AN, mean growth hormone (GH) was negatively correlated with global pituitary volume (R = 0.31, p = 0.031) and APV (R = 0.29, p = 0.037). Absolute and relative undernutrition led to a decreased anterior pituitary gland volume, which was reversible with weight gain, correlated with low bodyweight, and blockade of gonadal hypothalamic-pituitary axis. Intriguing inverse correlation between anterior pituitary gland volume and GH plasma level could suggests a low storage capacity of anterior pituitary gland and increased reactivity to low insulin-like growth factor type 1.