Rb optically pumped magnetometer (OPM) based on electron paramagnetic resonance (EPR) show advantages to measure the nuclear magnetization and have succeeded in fundamental physics and rotation sensing, etc. The magnetometry sensitivity is a key performance of these Rb OPMs which should be improved. In this study, a high sensitivity Rb OPM is demonstrated theoretically and experimentally. To improve the sensitivity, acousto-optic modulation based on balanced detection is applied to suppress the probe noises. Compared with the conventional optical rotation detection for this OPM configuration, the probe noise shows a significant suppression especially in low frequencies. Eventually, a simultaneous dual-axis transverse measurement with 30 fT/Hz1/2 sensitivity is achieved in a 200 Hz bandwidth and a 250nT linear working range. In addition, we utilize a closed-loop feedback to improve the stability and enlarge the transverse measurement range to 10µT order of magnitude while maintain the open-loop performances. A quasi-static magnetic field measurement can also be achieved in the longitudinal direction in the closed-loop mode. This OPM can serve for the nuclear magnetization measurement with a high sensitivity especially in environments with a large magnitude of the external magnetic field.