2D Fourier Transform on a non-rectangular space

Is it possible to perform a Fourier transform on a shape instead of a rectangular region? To be specific I am attempting to make a linear zoom function that doesn't produce any pixelation and that mimics natural blur that occurs with distance. After sketching the problem in Blender it seems that the best approach would be to utilize a frustum from an ovular cone, generate a Fourier transform from the an oval-bound section of the image, and extend the transform with additional null high frequencies to produce multiple pixels from a single pixel (or strip off high frequencies for zoom-out). Finally I would apply an inverse transform. This operation would be performed on a pixel-by-pixel basis for each pixel of the final image. As good as this may sound I'm not familiar with a Fourier transform on anything other than rectangular space. Can anyone suggest a good place to start?



Note: Computing power is not an issue for this problem as the system will be parellelized across 4096 cores at ~ 1mhz each for a total of nearly 4 teraflops of computing power using 2x 280X Radeon GPU's.