Portable LLMs with llamafile

Posted May 15, 2024 7:55 UTC (Wed) by taladar (subscriber, #68407)
Parent article: Portable LLMs with llamafile

My impression so far is that LLMs and other generative AI are mainly made complicated by the fact that it all involves Python in some way and the AI ecosystem seems to have copied the messiness of Python distribution while also lagging a few versions behind. If tooling could be improved in that respect, ideally by completely getting rid of any Python requirements and any GPU specific libraries (as in "this is the pytorch for ROCm, this is the one for CUDA, this is the one for CPU") it would make it a lot more accessible.

Portable LLMs with llamafile

Posted May 15, 2024 16:55 UTC (Wed) by ben.alkov (guest, #171521) [Link]

Specifically addressing Torch - their recently-available binary distro downloadable from PyPi *does* bundle everything, so that there's no need to try to figure out which one you need. Of course, the download is significantly larger than the individual platform packages.

Portable LLMs with llamafile

Posted May 15, 2024 18:18 UTC (Wed) by geofft (subscriber, #59789) [Link]

I mean, there was/is a thing called just Torch, but PyTorch, a set of Python bindings to it, got popular because you do want some sort of REPL / interpreted language to work with it and Python is a pretty good one, especially for the problem domain of scientific computing + interfacing with data sets. (I think OG Torch came with Lua, which is a little easier to embed into a C codebase but much less popular for scientific computing.)

Would it be helpful if PyTorch were additionally available as a standalone application with an embedded Python interpreter (probably an embedded Jupyter or something) so it was one thing to download and install and it was independent of any other Python version/environment you might have on your aystem?