The Transformer from "Attention Is All You Need" is now the ubiquitous architecture in state-of-the-art models such as GPT and AlphaFold. At Transformer's heart and soul is the "attention mechanism". We propose an attention-based deep direct sampling method for solving Electrical Impedance Tomography, a class of boundary value inverse problems. We try to provide an example of whether and how one can benefit from the theoretical structure of a mathematical problem to develop task-oriented and structure-conforming neural operators. Then, inspired by the idea of employing partial differential equation (PDE) solvers as preconditioners for this inverse problem, we investigate a "solver-in-the-loop" training mechanism for Transformer-based neural operators. It allows the interaction between learnable parameters in a PDE solver module and those in neural networks for image reconstruction. The new method achieves superior reconstruction quality compared with its predecessors and is robust to noise.