We introduce a novel approach to tackle iconic linear map- ping between two images. We adopt a grid-based parametriza- tion of the deformation field that is encoded by a higher order graphical model. In the proposed formulation, latent vari- ables correspond to local grid displacement vectors and unary potentials locally quantify the level of alignment between the two images. Higher order constraints that involve third and forth order potentials, enforce the linearity of the resulting transformation. The resulting formulation is modular with respect to the image metric used to evaluate the correctness of mapping as well as with respect to the nature of the linear transformation (rigid, similarity, or affine). Inference on this graph is performed through dual decomposition. Compari- son with classic algorithms demonstrates the potential of our approach.