|
Forum Index : Microcontroller and PC projects : Running AI models on your own computer: a detailed investigation
| Author | Message | ||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 12242 |
https://mather.host/ai/local-models/ Local language models Which AI models can you run on your own hardware that are actually useful for embedded C and MMBasic work? Ten models were tested on a workstation with a 20 GB graphics card, on real tasks: writing C that is checked under AddressSanitizer, answering questions about the MMBasic source, and extending a BASIC interpreter. The write-up covers the method, the results, how to set up Ollama, and what more money buys, from a used 24 GB card to a 128 GB Ryzen AI Max+ machine. |
||||
| homa Guru Joined: 05/11/2021 Location: GermanyPosts: 771 |
Peter, Very interesting—thanks for sharing the AI research and findings. Cool site overall! Was it made with AI? Matthias |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 12242 |
No. I did all myself in an hour (NOT) |
||||
| robert.rozee Guru Joined: 31/12/2012 Location: New ZealandPosts: 2562 |
hi Peter, you wrote: when considering a local setup, one should probably also factor in electricity costs. here in new zealand i am paying 25 cents per unit (kWh) of electricity. if i were to run a home machine that consumed 1kW continuously for a month - ie, working non-stop - that would add up to around nz$180 for the electricity alone. even more extreme, a fellow over on the eevblog forums put together a setup that i estimate would be consuming around 10kW when running, ie nz$2500/month if working non-stop. the forum discussion (from page 14 onwards) is worth a read, see: https://www.eevblog.com/forum/testgear/eevblog-121gw-multimeter-firmware-details/325/ so if you factor in home electricity cost, how much does that offset the cost of a $200/month plan? cheers, rob :-) Edited 2026-11-15 23:12 by robert.rozee |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 12242 |
True, I'll add something to the analysis. The card I use is the RTX 4000 ADA SFF which has 20GB of VRAM and a max power draw of 70W. Electricity would certainly be different with a pair of RTX4090s at 450W each. Of course, even with them you are unlikely to run a local environment 24/7 |
||||
| Bleep Guru Joined: 09/01/2022 Location: United KingdomPosts: 885 |
Hi Peter, I'm trying to run your bench tests on my lowly laptop. Asking a general question of gemma 4 26b I get 12 to 14 Tps selftest works and passes, but the other test seem to fail, any ideas? { "when": "2026-11-16 19:52", "os": "Windows-11-10.0.26200-SP0", "cpu": "AMD Ryzen 7 5825U with Radeon Graphics", "ollama": "0.40.16", "ollama_env": { "OLLAMA_FLASH_ATTENTION": "1", "OLLAMA_KEEP_ALIVE": "30m", "OLLAMA_KV_CACHE_TYPE": "q8_0", "OLLAMA_NUM_PARALLEL": "1" }, "gpu": "", "ollama_server": { "OLLAMA_FLASH_ATTENTION": "true", "OLLAMA_KV_CACHE_TYPE": "q8_0", "OLLAMA_NUM_PARALLEL": "1", "OLLAMA_CONTEXT_LENGTH": "0", "OLLAMA_KEEP_ALIVE": "30m0s", "OLLAMA_MAX_LOADED_MODELS": "0" }, "ram": "32 GB @ 3200 MT/s" python selftest_hard.py printf good=PASS bad=FAIL (correct) FAIL multi got [-5-ab-ff-Q- 07|x |%|00000ABC] want [-5-ab-ff-Q- 07|x expr good=PASS bad=FAIL (correct) FAIL [6 AND 3 OR 8] got 6 err 0 want 10 err 0 } python selftest_real.py basic reference PASS unchanged fails: FAIL case 5 (pass 1): returned 20 want 110 RETURN keeps inner FOR loops fails: FAIL case 17 (pass 1): returned 130 want 0 FOR/NEXT ignore the GOSUB boundary fails: FAIL case 18 (pass 1): returned 0 want 110 skip test ignores the step sign fails: FAIL case 7 (pass 1): returned 0 want 0 NEXT always tests <= fails: FAIL case 7 (pass 1): returned 0 want 0 GOSUBs and FORs share 8 slots fails: FAIL case 15 (pass 1): returned 110 want 0 GOSUB stack not reset fails: FAIL case 14 (pass 1): returned 20 want 0 progstore reference PASS both bugs fails: ==3619==ERROR: AddressSanitizer: global-buffer-overflow on address 0x5af172e159c0 at pc 0x71ca73728854 bp 0x7f memmove length only fails: ==3632==ERROR: AddressSanitizer: global-buffer-overflow on address 0x5f9191ef4940 at pc 0x776e13328854 bp 0x7f capacity check only fails: FAIL op 176: prog_store(65, 41 chars) returned -1 want 0 (1004 bytes used before) heap reference PASS free runs into the next block fails: FAIL op 33: heap_free_bytes 4608 want 4096, heap_largest 2816 want 2816 move frees the old block first fails: FAIL op 323: heap_realloc(293) returned offset 5376 want -1 (-1 = NULL) grow in place does not zero fails: FAIL op 139: heap_realloc(233 -> 284) new pages not zeroed first fit off by one page fails: FAIL op 0: heap_alloc(6829) returned offset 256 want 0 (-1 = NULL) selftest OK Regards Kevin. |
||||
| dMajo Newbie Joined: 18/05/2020 Location: ItalyPosts: 31 |
To run local models the way to go is to dedicate an Mac Studio M5Max w 128GB of unified ram (or M5Ultra & 256GB) to the task an use your PC/Workstation as client, without investing in gpu hw upgrades. ALLdataEE.com EDU Promos |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 12242 |
Think I'll buy a couple of these. Should be perfect for PicoMite development |
||||
| Mixtel90 Guru Joined: 05/10/2019 Location: United KingdomPosts: 9373 |
But you don't like Linux... That's put you off, hasn't it? :) Mick Zilog Inside! nascom.info for Nascom & Gemini |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4401 |
Maybe get half a dozen to round the price up nicely. John |
||||
| The Back Shed's forum code is written, and hosted, in Australia. | © JAQ Software 2026 |