Further progress in constructing highly accurate density functionals by enforcing known laws of interelectron interactions is slow, so fitting techniques are usually employed nowadays. These approaches were shown to lead to overfitting when a functional becomes unreliable for properties on which it was not trained on. An approach to maintain the correct physical behavior of a functional during its training is required to build more complex and accurate functionals, including those based on neural networks. We devise such an approach and apply it to reparameterize the heavily fitted and popular M06-2X functional on its original training set. The resulting physics-informed functionals piM06-2X and piM06-2X-DL approached the accuracy of M06-2X in thermochemical tasks and the accuracy of PBE0 in electron densities, taking the best out of both worlds. Surprisingly, we find that a very similar performance can be achieved directly by using the PBE-2X functional without any fitting. The proposed approach should be indispensable for training future neural-network-based functionals.