If it’s for a single low frequency workflow that GPU is enough, but you will be limited to small models which are mostly useless unless fine-tuned for your use case. If it’s serving users for the entire company with a big enough model to be useful you would need 192-384GB of VRAM. So a server between $20k and $40k.
The server will require maintenance, and somebody will have to develop the workflow and integration with your data.
It’s also important to know what they want to do, a basic embedding model for semantic search would work, agents not so much.
It really depends on the camera you have and how you use it. If it’s high enough megapixels (20+) and you bought it for high fps burst shot (needed to shoot moving subjects or kids), you might need a v90 cards which will be 4 times that price. Otherwise the Canvas is ok, but these days you can find them in the same price range of Prograde which would be my suggestion. I have and use both.
None of what this does is new, the issue is that’s being commodified: in a short amount of time these techniques will be available to non experts in the field.
Those values are most of what I was looking for. An LLM is just predicting the next token (for simplicity, a word). It does so by generating every possible word with a probability associated with it, and then picking a random word from this list, influenced by its probability. So for the sentence “a cat sat” it might generate “on: 0.6”, “down: 0.2”, and so on. 0.6 just means 60%, and all the values add up to 1 (100%). Now, the number of tokens generated can be as big as the context, so you might want to pick randomly from the top 10, you control this with the parameter top_k, or you might want to discard all the words below 20%, you control this with min_p. And finally, in cases where you have a token with a big probability followed by tokens with very low probability, you might want to squash these probabilities to be closer together, by decreasing the higher tokens and increasing the lower tokens. You control this with the temperature parameter where 0.1 is very little squashing, and 1 a lot of it. In layman terms this is the amount of creativity of your model. 0 is none, 1 is a lot, 2 is mentally insane.
Now, without knowing your hardware or why you need docker, it is hard to suggest a tool to run LLMs. I am not familiar with what you are using, but it seems to not be maintained, and likely lacks the features needed for a modern LLM to work properly.
For consumer grade hardware and personal use, the best tool these days is llamacpp, usually through a newbie friendly wrapper like LMStudio which support other backends as well and provide so much more than just a UI to download and run models. My advice is to download it and start there (it will download the right backend for you, so no need to install anything else manually).
I agree, it is a very expensive hobby, and it gets decent in the range 30-80b. However, the model you are using should not perform that bad, it seems that you might be hitting a config issue. Would you mind sharing the cli command you use to run it?
This is with the extra brightness and contrast added by the iOS screenshot functionality during screenshot. I would need a second phone to truly show you the horror of liquid
I don’t dislike aero or glass, I just dislike iOS implementation of it. With the wrong background sometimes you can’t read buttons for example, and a myriad of little things that are insignificant on their own but a pain in numbers.
I bet the Iranian hack only released non compromising material as a way to scare/extort him and now he is on edge.
If it’s for a single low frequency workflow that GPU is enough, but you will be limited to small models which are mostly useless unless fine-tuned for your use case. If it’s serving users for the entire company with a big enough model to be useful you would need 192-384GB of VRAM. So a server between $20k and $40k.
The server will require maintenance, and somebody will have to develop the workflow and integration with your data.
It’s also important to know what they want to do, a basic embedding model for semantic search would work, agents not so much.
The harness is as important as the model
It really depends on the camera you have and how you use it. If it’s high enough megapixels (20+) and you bought it for high fps burst shot (needed to shoot moving subjects or kids), you might need a v90 cards which will be 4 times that price. Otherwise the Canvas is ok, but these days you can find them in the same price range of Prograde which would be my suggestion. I have and use both.
Exactly where people who need to read this message are. What’s your point?
None of what this does is new, the issue is that’s being commodified: in a short amount of time these techniques will be available to non experts in the field.
Those values are most of what I was looking for. An LLM is just predicting the next token (for simplicity, a word). It does so by generating every possible word with a probability associated with it, and then picking a random word from this list, influenced by its probability. So for the sentence “a cat sat” it might generate “on: 0.6”, “down: 0.2”, and so on. 0.6 just means 60%, and all the values add up to 1 (100%). Now, the number of tokens generated can be as big as the context, so you might want to pick randomly from the top 10, you control this with the parameter top_k, or you might want to discard all the words below 20%, you control this with min_p. And finally, in cases where you have a token with a big probability followed by tokens with very low probability, you might want to squash these probabilities to be closer together, by decreasing the higher tokens and increasing the lower tokens. You control this with the temperature parameter where 0.1 is very little squashing, and 1 a lot of it. In layman terms this is the amount of creativity of your model. 0 is none, 1 is a lot, 2 is mentally insane.
Now, without knowing your hardware or why you need docker, it is hard to suggest a tool to run LLMs. I am not familiar with what you are using, but it seems to not be maintained, and likely lacks the features needed for a modern LLM to work properly. For consumer grade hardware and personal use, the best tool these days is llamacpp, usually through a newbie friendly wrapper like LMStudio which support other backends as well and provide so much more than just a UI to download and run models. My advice is to download it and start there (it will download the right backend for you, so no need to install anything else manually).
I agree, it is a very expensive hobby, and it gets decent in the range 30-80b. However, the model you are using should not perform that bad, it seems that you might be hitting a config issue. Would you mind sharing the cli command you use to run it?
You are playing with ancient stuff that wasn’t even good at release. Try these:
A 4b model performing like a 30b model: https://huggingface.co/Nanbeige/Nanbeige4.1-3B
Google open source version of Gemini: https://huggingface.co/google/gemma-3-4b-it
This is with the extra brightness and contrast added by the iOS screenshot functionality during screenshot. I would need a second phone to truly show you the horror of liquid
I don’t dislike aero or glass, I just dislike iOS implementation of it. With the wrong background sometimes you can’t read buttons for example, and a myriad of little things that are insignificant on their own but a pain in numbers.
Yeah, that’s the same feeling I have. In the past it was “I don’t like it but can get used to it”, now it’s so bad it’s borderline unusable.
Everything is better than Liquid Glass
Cloudflare tunnel: tailscale
What do you mean? The entire stack is open source.
It’s misleading and dishonest to frame this as “what Brussels says”, it’s the other 26 that don’t have a Russian puppet in power saying it.
He said tough but he meant hard.
“They do this because it goes unpunished, they are not getting arrested” says guy on a video showing multiple people being arrested.