High-precision mass measurements of exotic ^{95-97}Ag isotopes close to the N=Z line have been conducted with the JYFLTRAP double Penning trap mass spectrometer, with the silver ions produced using the recently commissioned inductively heated hot cavity catcher laser ion source at the Ion Guide Isotope Separator On-Line facility. The atomic mass of ^{95}Ag was directly determined for the first time. In addition, the atomic masses of β-decaying 2^{+} and 8^{+} states in ^{96}Ag have been identified and measured for the first time, and the precision of the ^{97}Ag mass has been improved. The newly measured masses, with a precision of ≈1 keV/c^{2}, have been used to investigate the N=50 neutron shell closure, confirming it to be robust. Empirical shell-gap and pairing energies determined with the new ground-state mass data are compared with the state-of-the-art ab initio calculations with various chiral effective field theory Hamiltonians. The precise determination of the excitation energy of the ^{96m}Ag isomer in particular serves as a benchmark for ab initio predictions of nuclear properties beyond the ground state, specifically for odd-odd nuclei situated in proximity to the proton dripline below ^{100}Sn. In addition, density functional theory calculations and configuration-interaction shell-model calculations are compared with the experimental results. All theoretical approaches face challenges to reproduce the trend of nuclear ground-state properties in the silver isotopic chain across the N=50 neutron shell and toward the proton dripline.