Transcriptional activation of estrogen receptor alpha (ERalpha) is regulated by the ligand-dependent activation function 2 and the constitutively active N-terminal activation function 1. To identify ERalpha N-terminal-specific coregulators, we screened a breast cDNA library by T7 phage display and isolated histone deacetylase 4 (HDAC4). HDAC4 interacts with the ERalpha N terminus both in vitro and in vivo. Presence of the ERalpha DNA binding domain and hinge region reduce HDAC4 recruitment whereas full-length ERalpha enhances recruitment. HDAC4 interaction is selective for the ERalpha and not ERbeta N terminus and occurs in the nucleus. We demonstrate in vivo that HDAC4 is recruited by the N terminus to the promoter of an endogenous estrogen responsive gene. HDAC4 suppresses transcriptional activation of ERalpha by estrogen and selective ER modulators (SERMs) such as tamoxifen in a cell type-dependent manner. Consistently, silencing of HDAC4 promotes the agonist effect of SERMs (tamoxifen and raloxifene) in a cell type-specific manner. These findings indicate a role for HDAC4 in regulating ERalpha activity as a novel N-terminal coregulator and uncover a mechanism by which certain cell types regulate SERM behavior.