I've been following the work that went into this video for a couple of months and have grown to love Level1Techs.
Check out their forum and especially Ubergarm
AI and You Against the Machine: Guide so you can own Big AI and Run Local - YouTube
https://www.youtube.com/watch?v=T17bpGItqXw
5 Comments
TheLeadenSea@sh.itjust.works · 2 pts · 1y
Ok, what machine is it? DL;dw
panda_abyss@lemmy.ca · 3 pts · 1y
Their setup is here:
https://forum.level1techs.com/t/full-deepseek-q1-with-the-ik-version-of-llama-cpp-on-am5-no-distills-just-a-quant/233530
TheLeadenSea@sh.itjust.works · 8 pts · 1y
Oh, so an NVIDIA card with 24GB VRAM
panda_abyss@lemmy.ca · 7 pts · 1y
Yep, and 2x64gb RAM
brucethemoose@lemmy.world · 2 pts · 1y
This is an excellent guide, way above most YT guides. ik_llama.cpp is a small miracle, and its a crime so few know of it.