Polarization and wavelength multiplexed metalenses address the bulkiness of traditional imaging systems. However, despite progress with numerical simulations and parameter scanning, the engineering complexity of classical methods highlights the urgent need for efficient deep learning approaches. This paper introduces a deep learning-driven inverse design model for polarization-multiplexed metalenses, employing propagation phase theory alongside spectral transfer learning to address chromatic dispersion challenges. The model facilitates the rapid design of metalenses with off-axis and dual-focus capabilities within a single wavelength. Numerical simulations reveal a focal length deviation of less than 5% and an average focusing efficiency of 43.3%. The integration of spectral transfer learning streamlines the design process, enabling multifunctional metalenses with enhanced full-color imaging and displacement measurement, thus advancing the field of metasurfaces.