Giant arcs in strong lensing galaxy clusters can provide a purely geometric determination of cosmological parameters, such as the dark energy density and equation of state. We investigate sources of noise in cosmography with giant arcs, focusing in particular on errors induced by density fluctuations along the line of sight and errors caused by modeling uncertainties. We estimate parameter errors in two independent ways, first by developing a Fisher matrix formalism for strong lensing parameters and next by directly ray-tracing through N-body simulations using a multiplane lensing code. We show that for reasonable power spectra, density fluctuations from large-scale structures produce >100% errors in cosmological parameters derived from any single sight line, precluding the use of individual clusters or "golden lenses" to derive accurate cosmological constraints. Modeling uncertainties can similarly lead to large errors, and we show that the use of parameterized mass models in fitting strong lensing clusters can significantly bias the inferred cosmological parameters. We lastly speculate on the means by which these errors may be corrected.