Fluorine (F) substitution is a common method of drug discovery and development. However, there are no accurate approaches available for predicting the bioactivity changes after F-substitution, as the effect of substitution on the interactions between compounds and proteins (CPI) remains a mystery. In this study, we constructed a data set with 111,168 pairs of fluorine-substituted and nonfluorine-substituted compounds. We developed a multimodal deep learning model (F-CPI). In comparison with traditional machine learning and popular CPI task models, the accuracy, precision, and recall of F-CPI (∼90, ∼79, and ∼45%) were higher than those of GraphDTA (∼86, ∼58, and ∼40%). The application of the F-CPI for the structural optimization of hit compounds against SARS-CoV-2 3CLpro by F-substitution achieved a more than 100-fold increase in bioactivity (IC50: 0.23 μM vs 28.19 μM). Therefore, the multimodal deep learning model F-CPI would be a veritable and effective tool in the context of drug discovery and design.