Home
JAQForum Ver 24.01
Log In or Join  
Active Topics
Local Time 12:15 18 Nov 2026 Privacy Policy
Jump to

Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me.

Forum Index : Microcontroller and PC projects : Running AI models on your own computer: a detailed investigation

Author Message
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 12242
Posted: 06:17pm 14 Nov 2026
Copy link to clipboard 
Print this post

https://mather.host/ai/local-models/

Local language models

Which AI models can you run on your own hardware that are actually useful for embedded C and MMBasic work? Ten models were tested on a workstation with a 20 GB graphics card, on real tasks: writing C that is checked under AddressSanitizer, answering questions about the MMBasic source, and extending a BASIC interpreter.

The write-up covers the method, the results, how to set up Ollama, and what more money buys, from a used 24 GB card to a 128 GB Ryzen AI Max+ machine.
 
homa

Guru

Joined: 05/11/2021
Location: Germany
Posts: 771
Posted: 07:52pm 14 Nov 2026
Copy link to clipboard 
Print this post

Peter,

Very interesting—thanks for sharing the AI research and findings. Cool site overall! Was it made with AI?

Matthias
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 12242
Posted: 07:59pm 14 Nov 2026
Copy link to clipboard 
Print this post

  Quote  Was it made with AI?

No. I did all myself in an hour (NOT)
 
robert.rozee
Guru

Joined: 31/12/2012
Location: New Zealand
Posts: 2562
Posted: 01:08pm 15 Nov 2026
Copy link to clipboard 
Print this post

hi Peter,
   you wrote:
  Quote  Even £10,000 of hardware doesn't buy a top cloud model; the largest open models need hundreds of gigabytes. The same money pays for the most expensive individual coding plans, about $200 a month, for over four years. Local models win on privacy, working offline and having no running costs, and they're good for routine work.


when considering a local setup, one should probably also factor in electricity costs. here in new zealand i am paying 25 cents per unit (kWh) of electricity. if i were to run a home machine that consumed 1kW continuously for a month - ie, working non-stop - that would add up to around nz$180 for the electricity alone.

even more extreme, a fellow over on the eevblog forums put together a setup that i estimate would be consuming around 10kW when running, ie nz$2500/month if working non-stop. the forum discussion (from page 14 onwards) is worth a read, see:
https://www.eevblog.com/forum/testgear/eevblog-121gw-multimeter-firmware-details/325/

so if you factor in home electricity cost, how much does that offset the cost of a $200/month plan?


cheers,
rob   :-)
Edited 2026-11-15 23:12 by robert.rozee
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 12242
Posted: 01:15pm 15 Nov 2026
Copy link to clipboard 
Print this post

True, I'll add something to the analysis. The card I use is the RTX 4000 ADA SFF which has 20GB of VRAM and a max power draw of 70W. Electricity would certainly be different with a pair of RTX4090s at 450W each. Of course, even with them you are unlikely to run a local environment 24/7
 
Bleep

Guru

Joined: 09/01/2022
Location: United Kingdom
Posts: 885
Posted: 08:15pm 16 Nov 2026
Copy link to clipboard 
Print this post

Hi Peter,
I'm trying to run your bench tests on my lowly laptop.
Asking a general question of gemma 4 26b I get 12 to 14 Tps
selftest works and passes, but the other test seem to fail, any ideas?
{
"when": "2026-11-16 19:52",
"os": "Windows-11-10.0.26200-SP0",
"cpu": "AMD Ryzen 7 5825U with Radeon Graphics",
"ollama": "0.40.16",
"ollama_env": {
 "OLLAMA_FLASH_ATTENTION": "1",
 "OLLAMA_KEEP_ALIVE": "30m",
 "OLLAMA_KV_CACHE_TYPE": "q8_0",
 "OLLAMA_NUM_PARALLEL": "1"
},
"gpu": "",
"ollama_server": {
 "OLLAMA_FLASH_ATTENTION": "true",
 "OLLAMA_KV_CACHE_TYPE": "q8_0",
 "OLLAMA_NUM_PARALLEL": "1",
 "OLLAMA_CONTEXT_LENGTH": "0",
 "OLLAMA_KEEP_ALIVE": "30m0s",
 "OLLAMA_MAX_LOADED_MODELS": "0"
},
"ram": "32 GB @ 3200 MT/s"

python selftest_hard.py
printf   good=PASS  bad=FAIL (correct) FAIL multi got [-5-ab-ff-Q-   07|x     |%|00000ABC] want [-5-ab-ff-Q-   07|x
expr     good=PASS  bad=FAIL (correct) FAIL [6 AND 3 OR 8] got 6 err 0 want 10 err 0
}

python selftest_real.py
basic      reference PASS
          unchanged                              fails: FAIL case 5 (pass 1): returned 20 want 110
          RETURN keeps inner FOR loops           fails: FAIL case 17 (pass 1): returned 130 want 0
          FOR/NEXT ignore the GOSUB boundary     fails: FAIL case 18 (pass 1): returned 0 want 110
          skip test ignores the step sign        fails: FAIL case 7 (pass 1): returned 0 want 0
          NEXT always tests <=                   fails: FAIL case 7 (pass 1): returned 0 want 0
          GOSUBs and FORs share 8 slots          fails: FAIL case 15 (pass 1): returned 110 want 0
          GOSUB stack not reset                  fails: FAIL case 14 (pass 1): returned 20 want 0
progstore  reference PASS
          both bugs                              fails: ==3619==ERROR: AddressSanitizer: global-buffer-overflow on address 0x5af172e159c0 at pc 0x71ca73728854 bp 0x7f
          memmove length only                    fails: ==3632==ERROR: AddressSanitizer: global-buffer-overflow on address 0x5f9191ef4940 at pc 0x776e13328854 bp 0x7f
          capacity check only                    fails: FAIL op 176: prog_store(65, 41 chars) returned -1 want 0 (1004 bytes used before)
heap       reference PASS
          free runs into the next block          fails: FAIL op 33: heap_free_bytes 4608 want 4096, heap_largest 2816 want 2816
          move frees the old block first         fails: FAIL op 323: heap_realloc(293) returned offset 5376 want -1 (-1 = NULL)
          grow in place does not zero            fails: FAIL op 139: heap_realloc(233 -> 284) new pages not zeroed
          first fit off by one page              fails: FAIL op 0: heap_alloc(6829) returned offset 256 want 0 (-1 = NULL)
selftest OK


Regards Kevin.
 
dMajo

Newbie

Joined: 18/05/2020
Location: Italy
Posts: 31
Posted: 01:57am 17 Nov 2026
Copy link to clipboard 
Print this post

To run local models the way to go is to dedicate an Mac Studio M5Max w 128GB of unified ram (or M5Ultra & 256GB) to the task an use your PC/Workstation as client, without investing in gpu hw upgrades.
ALLdataEE.com  EDU Promos
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 12242
Posted: 12:45pm 17 Nov 2026
Copy link to clipboard 
Print this post

Think I'll buy a couple of these. Should be perfect for PicoMite development  
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 9373
Posted: 01:27pm 17 Nov 2026
Copy link to clipboard 
Print this post

But you don't like Linux...

That's put you off, hasn't it?  :)
Mick

Zilog Inside! nascom.info for Nascom & Gemini
 
JohnS
Guru

Joined: 18/11/2011
Location: United Kingdom
Posts: 4401
Posted: 02:23pm 17 Nov 2026
Copy link to clipboard 
Print this post

Maybe get half a dozen to round the price up nicely.

John
 
Print this page


To reply to this topic, you need to log in.

The Back Shed's forum code is written, and hosted, in Australia.
© JAQ Software 2026