We present a conservative spectral method for the fully nonlinear Boltzmann collision operator based on the weighted convolution structure in Fourier space developed by Gamba and Tharkabhushnanam.. This method can simulate a broad class of collisions, including both elastic and inelastic collisions as well as angularly dependent cross sections in which grazing collisions play a major role. The extension presented in this paper consists of factorizing the convolution weight on quadrature points by exploiting the symmetric nature of the particle interaction law, which reduces the computational cost and memory requirements of the method to O(M(2)N(4)logN) from the O(N-6) complexity of the original spectral method, where N is the number of velocity grid points in each velocity dimension and M is the number of quadrature points in the factorization, which can be taken to be much smaller than N. We present preliminary numerical results.