Alpaca is a neat little flatpak that containerizes everything and makes running local models so easy that I can literally do it without a mouse or keyboard.
Oh my god I feel so stupid. I’ve been arguing back and forth whether it was worth de-atomizing my steam deck to spin up alpaca in docker. I forgot they have a flatpak
Bazzite also has podman, though not specifically docker, in the core OS.
So... I have spun up one local LLM in Alpaca, told it what hardware, OS, and environment it is in/on, told it to generate a context prompt to inform itself of all that... and its now helping me try to figure out how/if it is possible to set up a podman container/environment... for LLMs that either Alpaca does not yet support, or I am too stupid to figure out.
Alpaca even has tools. You can give an LLM the ability to search the web for something, and find some info or what not.
ROCm on a Deck seems to kind of sort of work via ... basically you spoof your gpu id in the podman environment, and then... you would either hwve to do the ole allocate more ram to gpu thing, or attempt to edit the LLM's config and such, to try an run in a much lower than expected vram situation.
(WIP)
Presumably you could tell it to do a lot of things but that seems like a bad idea lol, anyway yeah, I was able to just tell it 'go online and lookup bazzite, familiarize yourself with pertinent details, reformulate context prompt.'
Most people do not have a local LLM in their pocket right now.
Most people have a client app that talks to a remote LLM, which 'lives' in an ecologically and economically dubious mega-datacenter, in their pocket right now.
Plenty of the AI functions on phones are on-device. I know the iPhone is capable of several text-based processing (summarizing, translating) offline, and they have an API for third party developers to use on-device models. And the Pixels have Gemini Nano on-device for certain offline functions.
That's so crazy that an LLM capable of doing anything at all can be that small! That's leaves room for like an entire .avi episode of family guy at dvd resolution on there, which is the natural choice for the remaining space of course
a 4k episode of family guy using H265 (HEVC) and assuming not too many cutaway gags could produce a file about 240MB. You could probably fit a 480i episode of south park in the remaining 60MB
Wow, just popped it onto my very slow desktop and this little model rips haha. I really think tiny LLMs with a good LoRA on top are going to be a huge deal going forward
You can fit text-only wikipedia on a normal Blu Ray as it's only about 24GB. You can also easily fit Llama 3.1 or any of the other open, offline capable ai models as they're only about 4GB.
I suggest the happy medium called Kiwix, directly from the programme you can download all of Wikipedia with medium-sized pictures for a hundred gigabytes or so.
500TB is still surprisingly reasonable for what is essentially a library of human (surface level) knowledge.
It would be interesting to know how large the file would be including all text form references (i'd imagine anything else such as videos would completely blow the proportions)
yes you really can; it's like 20-25 gb depending on how recent of a copy you have. I've been seeding wikipedia for almost a year and it barely takes any space on my computer
The full 2025-04 English-only ZIM dump is about 120 GB. That includes reduced-size images as well as all articles. I think the text-only version is in the 40-60 GB range.
There are smaller ZIM versions in the ~4 GB range that would fit on a DVD, but they're only a subset for specific topics or for a list of the most popular topics.
It's not the LLM that does the web searching, but the software stack around it. On its own, an LLM is just a text completer. What you'd need a frontend like OpenWebUI or Perplexica that would ask the LLM for, say five internet search queries that could return useful information for the prompt, throw those queries into SearxNG, and then pipe the results into the LLM's context for it to be used.
As for the models themselves, any decently-sized one that was released fairly recently would work. If you're looking specifically for open-source rather than open-weight models (meaning that the training data and methodologies were also released rather than just the model weights), GPT-OSS 20B/120B and the OLMo models are recent standouts there. If not, the Qwen3 series are pretty good. (There are other good models out there, this is just what I remember off the top of my head.)
53 Comments
DarkCloud@lemmy.world · 60 pts · 259d
You can get offline versions of LLMs.
criss_cross@lemmy.world · 14 pts · 259d
And gpt-oss is an offline version of chatgpt
utopianfiat@lemmy.world · 9 pts · 259d
Indeed https://huggingface.co/openai-community
linkinkampf19@lemmy.world · 7 pts · 259d
First thing that came to mind: GPT4All
sp3ctr4l@lemmy.dbzer0.com · 4 pts · 259d
I've been toying with Qwen3.
On my steam deck.
8 bil param model runs stably.
Its's opensource too!
Alpaca is a neat little flatpak that containerizes everything and makes running local models so easy that I can literally do it without a mouse or keyboard.
JustAnotherKay@lemmy.world · 2 pts · 255d
Oh my god I feel so stupid. I’ve been arguing back and forth whether it was worth de-atomizing my steam deck to spin up alpaca in docker. I forgot they have a flatpak
sp3ctr4l@lemmy.dbzer0.com · 1 pts · 255d
Bazzite also has podman, though not specifically docker, in the core OS.
So... I have spun up one local LLM in Alpaca, told it what hardware, OS, and environment it is in/on, told it to generate a context prompt to inform itself of all that... and its now helping me try to figure out how/if it is possible to set up a podman container/environment... for LLMs that either Alpaca does not yet support, or I am too stupid to figure out.
Alpaca even has tools. You can give an LLM the ability to search the web for something, and find some info or what not.
ROCm on a Deck seems to kind of sort of work via ... basically you spoof your gpu id in the podman environment, and then... you would either hwve to do the ole allocate more ram to gpu thing, or attempt to edit the LLM's config and such, to try an run in a much lower than expected vram situation.
(WIP)
Presumably you could tell it to do a lot of things but that seems like a bad idea lol, anyway yeah, I was able to just tell it 'go online and lookup bazzite, familiarize yourself with pertinent details, reformulate context prompt.'
SubArcticTundra@lemmy.ml · 2 pts · 259d
https://ollama.org/
Ghostalmedia@lemmy.world · 1 pts · 259d
I mean, most people have a local LLM in their pocket right now.
sp3ctr4l@lemmy.dbzer0.com · 4 pts · 258d
Unless I am missing something:
Most people do not have a local LLM in their pocket right now.
Most people have a client app that talks to a remote LLM, which 'lives' in an ecologically and economically dubious mega-datacenter, in their pocket right now.
GamingChairModel@lemmy.world · 3 pts · 258d
Plenty of the AI functions on phones are on-device. I know the iPhone is capable of several text-based processing (summarizing, translating) offline, and they have an API for third party developers to use on-device models. And the Pixels have Gemini Nano on-device for certain offline functions.
tetris11@feddit.uk · 1 pts · 258d
My phone does speech-to-text flawlessly offline, it's a crazy useful little LLM tool
sp3ctr4l@lemmy.dbzer0.com · 1 pts · 258d
Oh!
Well, I didn't know that.
I'm too poor to be able to afford such fancy phones.
Ghostalmedia@lemmy.world · 2 pts · 258d
Gemini nano, Apple Intelligence On-device, etc.
tomiant@piefed.social · 31 pts · 259d
FCKGW-RHQQ2-YXRKT-8TG6W-2B7Q8
Ghostalmedia@lemmy.world · 6 pts · 259d
CrAcKeD
eager_eagle@lemmy.world · 5 pts · 259d
make sure to disconnect the internet first
bjoern_tantau@swg-empire.de · 26 pts · 259d
It's just audio of French farting cats.
Lemmyoutofhere@lemmy.ca · 17 pts · 259d
Le pfffft.
Akasazh@feddit.nl · 2 pts · 258d
My bet was on porn.
Or a copy of an old Encarta cd-rom
SSUPII@sopuli.xyz · 20 pts · 259d
If we assume a CD, you can probably fit a 256M parameters model in it. But it will LOAD.
MacNCheezus@lemmy.today · 11 pts · 259d
DVDs exist. They can fit approx. 7B params, enough to be somewhat productive.
khepri@lemmy.world · 14 pts · 259d
Could you crunch an LLM into 700Mb that was still functional? Cause this looks like a fun thing to actually do as a joke.
Edit, I bet I could get https://huggingface.co/distilbert/distilgpt2 to run off a CD. How many tps am I gonna get guys 🤣
yellowbadbeast@lemmy.blahaj.zone · 11 pts · 259d
Qwen3-0.6B is about 400 MB at Q4 and is surprisingly coherent for what it is.
khepri@lemmy.world · 7 pts · 259d
That's so crazy that an LLM capable of doing anything at all can be that small! That's leaves room for like an entire .avi episode of family guy at dvd resolution on there, which is the natural choice for the remaining space of course
tetris11@feddit.uk · 2 pts · 258d
a 4k episode of family guy using H265 (HEVC) and assuming not too many cutaway gags could produce a file about 240MB. You could probably fit a 480i episode of south park in the remaining 60MB
khepri@lemmy.world · 2 pts · 259d
Wow, just popped it onto my very slow desktop and this little model rips haha. I really think tiny LLMs with a good LoRA on top are going to be a huge deal going forward
lime@feddit.nu · 4 pts · 259d
there's also tinyllama, which is somewhere around 600MB. it's hilariously inept. it's like someone jpeg-compressed a robot.
also you're only gonna load off of that cd once so it'll perform fine.
NullPointerException@lemmy.ca · 11 pts · 259d
That’s just Dr Sbaitso.
faizalr@piefed.social · 7 pts · 259d
It reminds me of the Britannica Encyclopedia on CD.
KyuubiNoKitsune@lemmy.blahaj.zone · 3 pts · 259d
Encarta 95
MidsizedSedan@lemmy.world · 7 pts · 259d
Isn't it possible to download all of wikipedia, and it being surprisenly a small file size? Can it fit on a CD?
AmbiguousProps@lemmy.today · 9 pts · 259d
It could fit on a BDXL disc.
masterspace@lemmy.ca · 4 pts · 259d
You can fit text-only wikipedia on a normal Blu Ray as it's only about 24GB. You can also easily fit Llama 3.1 or any of the other open, offline capable ai models as they're only about 4GB.
gustofwind@lemmy.world · 2 pts · 259d
could also store it on a flashdrive or micro sd card
SSUPII@sopuli.xyz · 6 pts · 259d
No
(English) 24,05GB without media. Adding media adds 428,36TB.
Axolotl_cpp@feddit.it · 3 pts · 259d
Can you give me the text only version link? I found only a version that is like 43gb
SSUPII@sopuli.xyz · 6 pts · 259d
The sizes I mentioned are from around 2023-2024, from https://en.wikipedia.org/wiki/Wikipedia:Size_of_Wikipedia
https://dumps.wikimedia.org/enwiki/ (https://en.wikipedia.org/wiki/Wikipedia:Database_download)
ZkhqrD5o@lemmy.world · 5 pts · 259d
I suggest the happy medium called Kiwix, directly from the programme you can download all of Wikipedia with medium-sized pictures for a hundred gigabytes or so.
J_on_LemmyML@lemmy.ml · 4 pts · 259d
GregorGizeh@lemmy.zip · 1 pts · 259d
500TB is still surprisingly reasonable for what is essentially a library of human (surface level) knowledge.
It would be interesting to know how large the file would be including all text form references (i'd imagine anything else such as videos would completely blow the proportions)
Axolotl_cpp@feddit.it · 5 pts · 259d
No, you really can't; It's like 43 gb the text only version
BanMe@lemmy.world · 2 pts · 259d
So gonna need like 2 CDs then
puppycat@lemmy.blahaj.zone · 1 pts · 259d
yes you really can; it's like 20-25 gb depending on how recent of a copy you have. I've been seeding wikipedia for almost a year and it barely takes any space on my computer
ptz@dubvee.org · 1 pts · 259d
The full 2025-04 English-only ZIM dump is about 120 GB. That includes reduced-size images as well as all articles. I think the text-only version is in the 40-60 GB range.
There are smaller ZIM versions in the ~4 GB range that would fit on a DVD, but they're only a subset for specific topics or for a list of the most popular topics.
rain_worl@lemmy.world · 0 pts · 259d
kiwix? that's compressed (afaik), and when i tried, it took up half of my disk space and needed ethernet
uriel238@lemmy.blahaj.zone · 6 pts · 259d
Offline LLMs exist but tend to have a few terabytes of base data just to get started (e.g. before LORAs)
nomorebillboards@lemmy.world · 10 pts · 259d
I thought it was more like 10-20GB to start out with a usable (but somewhat stupid) model.
Are you confusing the size of the dataset with the size of the model?
SanctimoniousApe@lemmings.world · 4 pts · 259d
Maybe they meant GTA?
SubArcticTundra@lemmy.ml · 4 pts · 259d
Does anyone know of any OSS LLMs that can search the web the way ChatGPT can?
yellowbadbeast@lemmy.blahaj.zone · 3 pts · 259d
It's not the LLM that does the web searching, but the software stack around it. On its own, an LLM is just a text completer. What you'd need a frontend like OpenWebUI or Perplexica that would ask the LLM for, say five internet search queries that could return useful information for the prompt, throw those queries into SearxNG, and then pipe the results into the LLM's context for it to be used.
As for the models themselves, any decently-sized one that was released fairly recently would work. If you're looking specifically for open-source rather than open-weight models (meaning that the training data and methodologies were also released rather than just the model weights), GPT-OSS 20B/120B and the OLMo models are recent standouts there. If not, the Qwen3 series are pretty good. (There are other good models out there, this is just what I remember off the top of my head.)
SubArcticTundra@lemmy.ml · 1 pts · 259d
Thank you
MonkderVierte@lemmy.zip · 2 pts · 259d
Depends. Does ChatGPT ignore robots.txt too?
cupcakezealot@piefed.blahaj.zone · 2 pts · 255d
anyone have the serial?