In Vienna rectifier, amplitude of the common mode voltage (CMV), fluctuation of the neutral point (NP) voltage, and harmonics of the input current are mutually coupled, making it difficult to balance these performances using the conventional strategy. A novel CMV-reduction modulation strategy based on the nearest four voltage vectors optimization is proposed, which can simultaneously decrease the NP voltage fluctuation and the input current harmonics. First, the issue of large switch-order harmonics in input current using the conventional strategy is analyzed according to the characteristics of the current ripple vector trajectory. The proposed strategy adopts four voltage vectors with low CMV amplitudes, which are nearest to the reference voltage vector. The duty cycles are distributed by the zero-sequence component to ensure that the average NP current is zero, which can reduce the NP voltage fluctuation and CMV simultaneously. Furthermore, to reduce switch-order harmonics and zero-crossing distortion of the input current, the optimal switching sequence is obtained by permutation and combination to minimize the vector synthesis error of the reference voltage. The duty cycles are optimized by limiting the range of zero-sequence component to eliminate the large voltage vector. Finally, the effectiveness of the proposed strategy is verified on a Vienna rectifier platform.