Signal Processing for Systems With Low Precision Quantization

In system design, digital signal processing involves operation upon and analysis of digital signals. The real physical signals are temporally or spatially varying physical quantities (for example, sound, electromagnetic radiation, images etc.) which can take values over an infinite, uncountable set. Digitizing physical signals is achieved in two steps sampling in the temporal or spatial domain followed by quantizing the samples to a finite set of values. For a bandlimited physical signal taking values over an infinite set, sampling, if done over a sufficiently fine grid, can be achieved with no loss of information. However, quantization is inherently a non-linear and lossy process. If the quantization precision is sufficiently large, the non-linearities can be ignored and signal processing algorithms for the system can be designed assuming no quantization i.e. samples are represented by their actual values. However, if the precision is low, the non-linearities become significant and it becomes necessary to take into account these effects while designing signal processing algorithms for such systems. In this thesis, we consider two such applications which require designing signal processing algorithms for low precision systems, viz. transceiver design for multiGigabit communication systems and mask design in optical lithography.