Apparatus and method for beamforming based on generalized eigen-analysis in multiple input multiple output wireless communication system

An apparatus and method for beamforming in a Multiple Input Multiple Output (MIMO) wireless communication system are provided. A Base Station (BS) includes a calculator for generating relational expressions between beamforming vectors of Mobile Stations (MSs) using channel information of each MS, for determining beamforming vector candidates of each MS from the relational expressions using generalized Eigen-analysis, and for selecting beamforming vectors for each MS from the beamforming vector candidates, and a plurality of Transmit (Tx) beamformers for performing Tx beamforming on dedicated pilot symbols to be transmitted to each MS using the beamforming vectors for each MS.