A series of 31 side-chain-modified analogs of cholesterol, zymosterol, lanosterol, and cycloartenol and the steroidal alkaloids solasodine and solanidine were studied as inhibitors of (S)-adenosyl-L-methionine:delta24(25)-sterol methyl transferase (SMT) enzyme activity from Saccharomyces cerevisiae. Two classes of sterol methylation inhibitors were tested: substrate analogs, including mechanism-based inhibitors, and transition state analogs. Several novel sterol methylation inhibitors that contained an aza, aziridine, or ammonium group in the sterol side chain were prepared and tested for the first time. The degree and kinetic pattern of methylation inhibition were found to be influenced by the position and nature of the variant functional group introduced into the side chain. The most potent inhibitors of SMT enzyme activity were transition state analog inhibitors (Ki values of 5 to 10 nM) that mimicked the structure and conformation of the natural substrate presumed to form in the ternary complex generated in the transition state. Steroidal alkaloids were potent competitive inhibitors with Ki values ranging from 2 to 30 microM, which is about the Kmapp of zymosterol, ca. 27 microM. An isosteric analog of the natural substrate, zymosterol, in which the 26/27-gem-dimethyl groups were joined to form a cyclopropylidene function is shown to be a potent irreversible mechanism-based inactivator of SMT enzyme activity that exhibits competitive-type inhibition, Ki 48 microM with a K(inact) of 1.52 min(-1). Mechanistic implications of these results provide new insights into the topology of the ternary complex involving sterol-AdoMet-enzyme.