MMSR: : Symbolic regression is a multi-modal information fusion task
Authors: 
Yanjie Li, Jingyi Liu, Min Wu, Lina Yu, Weijun Li, Xin Ning, Wenqiang Li, Meilan Hao, Yusong Deng, Shu WeiAuthors Info & Claims
Yanjie Li, Jingyi Liu, Min Wu, Lina Yu, Weijun Li, Xin Ning, Wenqiang Li, Meilan Hao, Yusong Deng, Shu WeiAuthors Info & ClaimsReferences
[1]
Wang Y., Wagner N., Rondinelli J.M., Symbolic regression in materials science, MRS Commun. 9 (3) (2019) 793––805.
[2]
Udrescu S.-M., Tegmark M., AI feynman: A physics-inspired method for symbolic regression, Sci. Adv. 6 (16) (2020) eaay2631,.
[3]
Udrescu S.-M., Tan A., Feng J., Neto O., Wu T., Tegmark M., AI feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity, Advances in Neural Information Processing Systems, vol. 33, Curran Associates, Inc., 2020, pp. 4860–4871.
[4]
Schmidt M., Lipson H., Distilling free-form natural laws from experimental data, Science 324 (5923) (2009) 81–85,. arXiv:https://www.science.org/doi/pdf/10.1126/science.1165893, URL https://www.science.org/doi/abs/10.1126/science.1165893.
[5]
Gustafson S., Burke E.K., Krasnogor N., On improving genetic programming for symbolic regression, 2005 IEEE Congress on Evolutionary Computation, vol. 1, IEEE, 2005, pp. 912–919.
[6]
Searson D.P., Leahy D.E., Willis M.J., GPTIPS: an open source genetic programming toolbox for multigene symbolic regression, Proceedings of the International Multiconference of Engineers and Computer Scientists, vol. 1, Citeseer, 2010, pp. 77–80.
[7]
Haider C., de França F.O., Burlacu B., Kronberger G., Shape-constrained multi-objective genetic programming for symbolic regression, Appl. Soft Comput. 132 (2023).
[8]
Mundhenk T., Landajuela M., Glatt R., Santiago C.P., Petersen B.K., et al., Symbolic regression via deep reinforcement learning enhanced genetic programming seeding, Adv. Neural Inf. Process. Syst. 34 (2021) 24912–24923.
[9]
B. He, Q. Lu, Q. Yang, J. Luo, Z. Wang, Taylor genetic programming for symbolic regression, in: Proceedings of the Genetic and Evolutionary Computation Conference, 2022, pp. 946–954.
[10]
Uy N.Q., Hoai N.X., O’Neill M., McKay R.I., Galván-López E., Semantically-based crossover in genetic programming: application to real-valued symbolic regression, Genet. Program. Evolvable Mach. 12 (2011) 91–119.
[11]
Jain M., Saihjpal V., Singh N., Singh S.B., An overview of variants and advancements of PSO algorithm, Appl. Sci. 12 (17) (2022) 8392.
[12]
Petersen B.K., Deep symbolic regression: Recovering mathematical expressions from data via policy gradients, 2019, CoRR abs/1912.04871, arXiv:1912.04871, URL http://arxiv.org/abs/1912.04871.
[13]
Alec R., Karthik N., Tim S., Ilya S., Improving Language Understanding by Generative Pre-Training, OpenAI, 2018.
[14]
Kim S., Lu P.Y., Mukherjee S., Gilbert M., Jing L., Čeperić V., Soljačić M., Integration of neural network-based symbolic regression in deep learning for scientific discovery, IEEE Trans. Neural Netw. Learn. Syst. 32 (9) (2021) 4166–4177,.
[15]
Mundhenk T.N., Landajuela M., Glatt R., Santiago C.P., Faissol D.M., Petersen B.K., Symbolic regression via neural-guided genetic programming population seeding, 2021, CoRR abs/2111.00053, arXiv:2111.00053, URL https://arxiv.org/abs/2111.00053.
[16]
La Cava W., Orzechowski P., Burlacu B., de França F.O., Virgolin M., Jin Y., Kommenda M., Moore J.H., Contemporary symbolic regression methods and their relative performance, 2021, arXiv preprint arXiv:2107.14351.
[17]
Li Y., Li W., Yu L., Wu M., Liu J., Li W., Hao M., Wei S., Deng Y., MetaSymNet: A dynamic symbolic regression network capable of evolving into arbitrary formulations, 2023, arXiv preprint arXiv:2311.07326.
[18]
Valipour V., You B., Panju M., Ghodsi A., SymbolicGPT: A generative transformer model for symbolic regression, 2021, CoRR abs/2106.14131, arXiv:2106.14131, URL https://arxiv.org/abs/2106.14131.
[19]
Luca B., Tommaso B., Alexander N., Aurelien L., Giambattista P., Neural symbolic regression that scales, in: Proceedings of the 38th International Conference on Machine Learning, in: Proceedings of Machine Learning Research, vol. 139, PMLR, 2021, pp. 936–945.
[20]
Vastl M., Kulhánek J., Kubalík J., Derner E., Babuska R., SymFormer: End-to-end symbolic regression using transformer-based architecture, 2022,. CoRR abs/2205.15764, arXiv:2205.15764, URL https://doi.org/10.48550/arXiv.2205.15764.
[21]
Kumar A., Vembu S., Menon A.K., Elkan C., Beam search algorithms for multilabel learning, Mach. Learn. 92 (2013) 65–89.
[22]
Liu D.C., Nocedal J., On the limited memory BFGS method for large scale optimization, Math. Program. 45 (1–3) (1989) 503–528.
[23]
Lee J., Lee Y., Kim J., Kosiorek A., Choi S., Teh Y.W., Set transformer: A framework for attention-based permutation-invariant neural networks, in: Proceedings of the 36th International Conference on Machine Learning, in: Proceedings of Machine Learning Research, vol. 97, PMLR, 2019, pp. 3744–3753. URL https://proceedings.mlr.press/v97/lee19d.html.
[24]
Chuang C.-Y., Robinson J., Lin Y.-C., Torralba A., Jegelka S., Debiased contrastive learning, Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 8765–8775.
[25]
Shojaee P., Meidani K., Barati Farimani A., Reddy C., Transformer-based planning for symbolic regression, Adv. Neural Inf. Process. Syst. 36 (2024).
[26]
Radford A., Kim J.W., Hallacy C., Ramesh A., Goh G., Agarwal S., Sastry G., Askell A., Mishkin P., Clark J., et al., Learning transferable visual models from natural language supervision, in: International Conference on Machine Learning, PMLR, 2021, pp. 8748–8763.
[27]
Jia C., Yang Y., Xia Y., Chen Y.-T., Parekh Z., Pham H., Le Q., Sung Y.-H., Li Z., Duerig T., Scaling up visual and vision-language representation learning with noisy text supervision, in: International Conference on Machine Learning, PMLR, 2021, pp. 4904–4916.
[28]
K. He, H. Fan, Y. Wu, S. Xie, R. Girshick, Momentum contrast for unsupervised visual representation learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9729–9738.
[29]
Chen T., Kornblith S., Norouzi M., Hinton G., A simple framework for contrastive learning of visual representations, International Conference on Machine Learning, PMLR, 2020, pp. 1597–1607.
[30]
Li J., Zhou P., Xiong C., Hoi S.C., Prototypical contrastive learning of unsupervised representations, 2020, arXiv preprint arXiv:2005.04966.
[31]
Li J., Xiong C., Hoi S.C., Mopro: Webly supervised learning with momentum prototypes, 2020, arXiv preprint arXiv:2009.07995.
[32]
Kim W., Son B., Kim I., Vilt: Vision-and-language transformer without convolution or region supervision, in: International Conference on Machine Learning, PMLR, 2021, pp. 5583–5594.
[33]
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C.L. Zitnick, D. Parikh, Vqa: Visual question answering, in: Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 2425–2433.
[34]
Wang Z., Yu J., Yu A.W., Dai Z., Tsvetkov Y., Cao Y., Simvlm: Simple visual language model pretraining with weak supervision, 2021, arXiv preprint arXiv:2108.10904.
[35]
Wang P., Yang A., Men R., Lin J., Bai S., Li Z., Ma J., Zhou C., Zhou J., Yang H., Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework, in: International Conference on Machine Learning, PMLR, 2022, pp. 23318–23340.
[36]
Piergiovanni A., Li W., Kuo W., Saffar M., Bertsch F., Angelova A., Answer-me: Multi-task open-vocabulary visual question answering, 2022, arXiv preprint arXiv:2205.00949.
[37]
A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, D. Kiela, Flava: A foundational language and vision alignment model, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 15638–15650.
[38]
Li J., Selvaraju R., Gotmare A., Joty S., Xiong C., Hoi S.C.H., Align before fuse: Vision and language representation learning with momentum distillation, Advances in Neural Information Processing Systems, vol. 34, 2021, pp. 9694–9705.
[39]
Li J., Li D., Xiong C., Hoi S., Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, in: International Conference on Machine Learning, PMLR, 2022, pp. 12888–12900.
[40]
Chen X., Djolonga J., Padlewski P., Mustafa B., Changpinyo S., Wu J., Ruiz C.R., Goodman S., Wang X., Tay Y., et al., PaLI-X: On scaling up a multilingual vision and language model, 2023, arXiv preprint arXiv:2305.18565.
[41]
Liu H., Li C., Wu Q., Lee Y.J., Visual instruction tuning, Advances in Neural Information Processing Systems, vol. 36, 2024.
[42]
Yu J., Wang Z., Vasudevan V., Yeung L., Seyedhosseini M., Wu Y., Coca: Contrastive captioners are image-text foundation models, 2022, arXiv, arXiv preprint arXiv:2205.01917.
[43]
Wang W., Bao H., Dong L., Bjorck J., Peng Z., Liu Q., Aggarwal K., Mohammed O.K., Singhal S., Som S., et al., Image as a foreign language: Beit pretraining for all vision and vision-language tasks, 2022, arXiv preprint arXiv:2208.10442.
[44]
Liu H., Li C., Wu Q., Lee Y.J., Visual instruction tuning, Advances in Neural Information Processing Systems, vol. 36, 2024.
[45]
Chang Y., Wang X., Wang J., Wu Y., Yang L., Zhu K., Chen H., Yi X., Wang C., Wang Y., et al., A survey on evaluation of large language models, ACM Trans. Intell. Syst. Technol. (2023).
[46]
Zhao W.X., Zhou K., Li J., Tang T., Wang X., Hou Y., Min Y., Zhang B., Zhang J., Dong Z., et al., A survey of large language models, 2023, arXiv preprint arXiv:2303.18223.
[47]
Touvron H., Lavril T., Izacard G., Martinet X., Lachaux M.-A., Lacroix T., Rozière B., Goyal N., Hambro E., Azhar F., et al., Llama: Open and efficient foundation language models, 2023, arXiv preprint arXiv:2302.13971.
[48]
Zeng A., Liu X., Du Z., Wang Z., Lai H., Ding M., Yang Z., Xu Y., Zheng W., Xia X., et al., Glm-130b: An open bilingual pre-trained model, 2022, arXiv preprint arXiv:2210.02414.
[49]
Ouyang L., Wu J., Jiang X., Almeida D., Wainwright C., Mishkin P., Zhang C., Agarwal S., Slama K., Ray A., et al., Training language models to follow instructions with human feedback, Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 27730–27744.
[50]
Arnaldo I., Krawiec K., O’Reilly U.-M., Multiple regression genetic programming, in: Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, Association for Computing Machinery, New York, NY, USA, 2014, pp. 879–886,.
[51]
McConaghy T., FFX: Fast, scalable, deterministic symbolic regression technology, Genetic Programming Theory and Practice IX, Springer, New York, NY, 2011, pp. 235–260,.
[52]
Nguyen S., Zhang M., Tan K.C., Surrogate-assisted genetic programming with simplified models for automated design of dispatching rules, IEEE Trans. Cybern. 47 (9) (2017) 2951–2965,.
[53]
Zhang F., Mei Y., Nguyen S., Zhang M., Evolving scheduling heuristics via genetic programming with feature selection in dynamic flexible job-shop scheduling, IEEE Trans. Cybern. 51 (4) (2021) 1797–1811,.
[54]
Y. Xu, Y. Liu, H. Sun, Reinforcement Symbolic Regression Machine.
[55]
Tibshirani R., Regression shrinkage and selection via the lasso, J. R. Stat. Soc. Ser. B Stat. Methodol. 58 (1) (1996) 267–288,.
[56]
Kamienny P.-A., d’Ascoli S., Lample G., Charton F., End-to-end symbolic regression with transformers, Adv. Neural Inf. Process. Syst. 35 (2022) 10269–10281.
[57]
W. Li, W. Li, L. Sun, M. Wu, L. Yu, J. Liu, Y. Li, S. Tian, Transformer-based model for symbolic regression via joint supervised learning, in: The Eleventh International Conference on Learning Representations, 2022.
[58]
Landajuela M., Lee C.S., Yang J., Glatt R., Santiago C.P., Aravena I., Mundhenk T., Mulcahy G., Petersen B.K., A unified framework for deep symbolic regression, Adv. Neural Inf. Process. Syst. 35 (2022) 33985–33998.
[59]
Liu J., Li W., Yu L., Wu M., Sun L., Li W., Li Y., SNR: Symbolic network-based rectifiable learning framework for symbolic regression, Neural Netw. 165 (2023) 1021–1034.
[60]
Holt S., Qian Z., van der Schaar M., Deep generative symbolic regression, 2023, arXiv preprint arXiv:2401.00282.
[61]
Browne C.B., Powley E., Whitehouse D., Lucas S.M., Cowling P.I., Rohlfshagen P., Tavener S., Perez D., Samothrakis S., Colton S., A survey of monte carlo tree search methods, IEEE Trans. Comput. Intell. AI games 4 (1) (2012) 1–43.
[62]
Li Y., Li W., Yu L., Wu M., Liu J., Li W., Hao M., Wei S., Deng Y., Discovering mathematical formulas from data via GPT-guided Monte Carlo tree search, 2024, arXiv preprint arXiv:2401.14424.
[63]
Meidani K., Shojaee P., Reddy C.K., Farimani A.B., SNIP: Bridging mathematical symbolic and numeric realms with unified pre-training, 2023, arXiv preprint arXiv:2310.02227.
[64]
Srivastava N., Hinton G., Krizhevsky A., Sutskever I., Salakhutdinov R., Dropout: A simple way to prevent neural networks from overfitting, J. Mach. Learn. Res. 15 (1) (2014) 1929–1958.
Index Terms
- MMSR: Symbolic regression is a multi-modal information fusion task
Index terms have been assigned to the content through auto-classification.
Recommendations
Real-time Emotion Pre-Recognition in Conversations with Contrastive Multi-modal Dialogue Pre-training
CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge ManagementThis paper presents our pioneering effort in addressing a new and realistic scenario in multi-modal dialogue systems called Multi-modal Real-time Emotion Pre-recognition in Conversations (MREPC). The objective is to predict the emotion of a forthcoming ...
Deep semi-supervised learning with contrastive learning and partial label propagation for image data
AbstractDeep semi-supervised learning is becoming an active research topic because it jointly utilizes labeled and unlabeled samples in training deep neural networks. Recent advances are mainly focused on inductive semi-supervised learning ...
Contrastive learning from label distribution: A case study on text classification
Highlights- The proposed method learns under the supervision of the predicted label distribution.
AbstractState-of-the-art text classification models are dominated by deep neural networks, but they still struggle to the issue of poor generalization ability when using cross entropy loss for training. One of the reasons is the training ...
Comments
Information & Contributors
Information
Published In
Elsevier B.V.
Publisher
Elsevier Science Publishers B. V.
Netherlands
Publication History
Published: 01 February 2025
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 08 Feb 2025