Developing lightweight training approaches for distributed machine learning is of key importance to deploy machine learning on Internet-of-Things (IoT) devices. Gossip SGD is attractive among distributed machine learning methods thanks to its applicability to various network topologies (irrespective of having a central entity) and simple parameters update strategy. However, it is known to be inefficient in non-homogeneous data distribution environments. Our work addresses this issue by introducing “parameters swapping” in the Gossip SGD during parameters update to let devices simply exchange their parameters. Our method, named Gossip Swap SGD, efficiently resolves a cause of insufficient accuracy or delayed convergence in parameters update for non-homogeneous data distribution. Our quantitative evaluation demonstrated that our method not only outperforms exiting methods in non-homogeneous data distribution but also has no degradation even in homogeneous data distribution.