c/linux_lugcast · by minnix@lemux.minnix.dev · 1yExploring Llamafile: Mozilla's Attempt in the World of Open Source AI https://itsfoss.com/llamafile/11 points · 4 comments · view on lemmy.world
4 Comments
muntedcrocodile@lemm.ee · 2 pts · 1y
The issue isnt if its easy to use the issue is a matter of compute. A majority of people on the internet only have mobile access. We need a way to let the masses utilise distributed compute in a secure and private way.
JackGreenEarth@lemm.ee · 1 pts · 1y
You can run many 7B models on phones with 8GB RAM
muntedcrocodile@lemm.ee · 3 pts · 1y
I find the 7b models dont have the capabilities id like and i cant image the tokens per second would be very good.
minnix@lemux.minnix.dev · 2 pts · 1y
Yeah running it and it actually being useful are two very different things.