The natural product icariin inhibits human phosphodiesterase-5 (PDE5) and represents a unique pharmacophore for treating erectile dysfunction, pulmonary hypertension, and other diseases. In this study, we explore the available icariin-derived chemical scaffolds through medicinal chemistry to develop novel icariin PDE5 inhibitors with improved potency and specificity. We synthesized six novel semi-synthetic icariin analogs as well as three naturally occurring icariin analogs, and characterized the structure-activity relationship in the context of human PDE5 inhibition using in vitro enzyme inhibition and kinetics assays and molecular modeling. Mammalian-cell-based assays and in vitro enzyme inhibition assays against human PDE6C further helped to identify the most potent and selective icariin analogs. Our results reveal the synergistic contribution of functional groups at the C3 and C7 positions of the icariin backbone towards PDE5 inhibition. Whereas a hydrophobic and flexible alkanol group at the C7 position is sufficient to enhance icariin analog potency, combining this group with a hydrophilic sugar group at the C3 position leads to further enhancement of potency and promotes specificity towards PDE5 versus PDE6C. In particular, compounds 3 and 7 exhibit Ki values of 0.036 ± 0.005 μM and 0.036 ± 0.007 μM towards PDE5 respectively, which are approaching those of commercial PDE5 inhibitors, and can effectively reduce GMP levels in cultured human BJ-hTERT cells. This study identifies novel icariin analogs as potent and selective PDE5 inhibitors poised to become lead compounds for further pharmaceutical development.