We report the accurate subarcsecond X-ray position of the new anomalous X-ray pulsar (AXP) XTE J1810-197, derived with a Chandra High Resolution Camera Target of Opportunity observation carried out in 2003 November. We also report the discovery of a likely IR counterpart based on a Very Large Telescope (IR band) Target of Opportunity observation carried out in 2003 October. Our proposed counterpart is the only IR source (Ks = 20.8) in the X-ray error circle. Its IR colors as well as the X-ray/IR flux ratio are consistent with those of the counterparts of all other AXPs (at variance with field star colors). Deep Gunn i-band images obtained at the 3.6 m ESO telescope detected no sources down to a limiting magnitude of 24.3. Moreover, we find that the pulsed fraction and count rates of XTE J1810-197 remained nearly unchanged since the previous Chandra and XMM-Newton observations (2003 August 27 and September 8, respectively). We briefly discuss the implications of these results. In particular, we note that the transient (or at least highly variable) nature of this AXP might imply a relatively large number of hidden members of this class.