Treatment optimization is usually formulated as an inverse problem, which starts with a prescribed dose distribution and obtains an optimized solution under the guidance of an objective function. The solution is a compromise between the conflicting requirements of the target and sensitive structures. In this paper, the treatment plan optimization is formulated as an estimation problem of a discrete and possibly nonconvex system. The concept of preference function is introduced. Instead of prescribing a dose to a structure (or a set of voxels), the approach prioritizes the doses with different preference levels and reduces the problem into selecting a solution with a suitable estimator. The preference function provides a foundation for statistical analysis of the system and allows us to apply various techniques developed in statistical analysis to plan optimization. It is shown that an optimization based on a quadratic objective function is a special case of the formalism. A general two-step method for using a computer to determine the values of the model parameters is proposed. The approach provides an efficient way to include prior knowledge into the optimization process. The method is illustrated using a simplified two-pixel system as well as two clinical cases. The generality of the approach, coupled with promising demonstrations, indicates that the method has broad implications for radiotherapy treatment plan optimization.