 Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me. |
Forum Index : Microcontroller and PC projects : better measurements by a running mean
Page 1 of 3   |
|||||
| Author | Message | ||||
andreas Senior Member  Joined: 07/12/2020 Location: GermanyPosts: 226 |
Hi all, I found that the measurements returned by sensors (temperature, humidity) or input devices like nunchuk controllers are 'jumping' around the 'real' physical values. If I do many measurements, add them and divide them by the number of measurements, then I have a chance to get the correct mean value for that physical property. He is a technique to get "better measurements" from the very beginning by calculating a running mean over the last 'n' measurements. n ist starting from 1 up to an choosen maximum of i.e. 100 or 1000. When n reaches the value of 1000 it will not grow any more. n defines a weight for the next measuremnt, so only a fraction of the difference of the old mean and the new measuremnt gets into the new mean. the longer the sequence is the lower is the weight for the next value. As an example I took a pico and measured the temperature by 'Pin(temp)'. The running mean M for the temperature is calculated and printed on every cycle of an endless loop. The minimum function 'Min' is not only increasing n by 1 at each cycle, it is limiting n by the maximum value 'Length'. Example in Picomite: ' running mean - get better measurements Const Length = 1000 ' up to Length values are counting Dim M As float = 0.0 ' running mean Dim n As integer = 0 ' counter [0..Length] While true do n = Min(Length, n + 1) M = M + (Pin(temp) - M ) / n Print M Pause 100 Loop If you replace the Pin(temp) by your own GPx input, you will certainly get better values than using only a single value. You can replace the While .. Loop by a For ..Next if you like but the strength of this method is within endless or continued measurements. - andreas (excuse my bad english, please) |
||||
phil99 Guru  Joined: 11/02/2018 Location: AustraliaPosts: 2640 |
Excellent, that is a simpler method than the one I was using I shall adopt it. |
||||
| Volhout Guru Joined: 05/03/2018 Location: NetherlandsPosts: 5089 |
hi andreas, Thank you for sharing this. I knew how it works, and have implemented it myself before, but my solution (as with phil99) was not as elegant as this one. This is much more elegant. Volhout. PicomiteVGA PETSCII ROBOTS |
||||
| andreas Senior Member Joined: 07/12/2020 Location: GermanyPosts: 226 |
Hello both phil99 & Volhout, thank you for the feedback - I'm glad if you can use it!  -andreas Edited 2021-12-17 18:54 by andreas |
||||
| Calli Regular Member  Joined: 20/10/2021 Location: GermanyPosts: 74 |
Hi Andreas, greets from Germany :) I like these little tips, thanks! Carsten |
||||
| andreas Senior Member Joined: 07/12/2020 Location: GermanyPosts: 226 |
Hi Carsten, you are wellcome!  -andreas |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4044 |
I think it effectively weights more recent readings less and less as time goes by - right? That may in effect make the result quite far from the wanted current value i.e. fail to track changes except with a bigger & bigger lag - right? If I'm right it's not suitable in various scenarios. In those cases EWMA (exponentially weighted moving average) may be appropriate. John Edited 2021-12-17 20:18 by JohnS |
||||
Mixtel90 Guru Joined: 05/10/2019 Location: United KingdomPosts: 7937 |
Won't that depend on how fast you want the readings, John? A fluctuating voltage monitor is a different case to something that has thermal lag, for example. Where you need a spot reading you are probably better taking a load of readings in quick succession, ignoring the first few for stability, and then averaging them. EWMA might be better at spotting trends though. Mick Zilog Inside! nascom.info for Nascom & Gemini Preliminary MMBasic docs & my PCB designs |
||||
CaptainBoing Guru Joined: 07/09/2016 Location: United KingdomPosts: 2170 |
it is effective. A bit of code to set a couple of pins high then read them as analogue in. The first column is the running mean, the second is simply whatever value is present at the time. The first column quickly settles down and within 10 reading is varying by only a couple of mV. the second is predictable all over the place as you would expect with an open input and no processing - all on unbuffered open lines, not terribly fair or even a good test. If I had more time I would put pots and small caps on both lines and see what it renders. I think, John, you are correct. I think it's ideal to run on the main thread just taking and averaging measurements for when they are needed, where things change moderately - temperature measurement (from analogue sources), battery voltages etc. Left to run for some time, it is very stable - the 100uV digit hardly changing ' running mean - get better measurements Const Length = 1000 ' up to Length values are counting Dim M As float = 0.0 ' running mean Dim n As integer = 0 ' counter [0..Length] Pin(27)=1 SetPin 27,DOUT SetPin 27,AIN Do n = Min(Length, n + 1) M = M + (Pin(27) - M ) / n Print M,Pin(27) Pause 100 Loop > > > RUN 0.662366 0.565591 0.582527 0.515591 0.553584 0.498925 0.541667 0.495161 0.535914 0.509677 0.529211 0.492473 0.526574 0.500538 0.523992 0.506452 0.520968 0.491936 0.520323 0.505914 0.518426 0.502151 0.516935 0.491936 0.516708 0.506452 0.515323 0.493548 0.514731 0.494624 0.514482 0.507527 0.513283 0.491936 0.513291 0.5 0.512337 0.495161 0.512312 0.503763 0.511802 0.503763 0.511168 0.489785 0.511267 0.505914 0.510663 0.497312 0.510366 0.494086 0.510463 0.508602 0.509857 0.494086 0.509908 0.497849 0.509807 0.508064 0.509283 0.489785 0.509383 0.504301 0.509056 0.502688 0.508781 0.492473 0.508966 0.508064 0.508602 0.494086 0.508527 0.494086 0.508573 0.507527 0.508206 0.491936 0.508285 0.498925 0.508172 0.506989 0.507894 0.491398 0.508026 0.505376 0.507764 0.502151 0.507625 0.489785 0.507766 0.506989 0.507492 0.493548 0.507527 0.497849 0.507572 0.507527 0.507318 0.49086 0.507398 0.504839 0.507242 0.502151 0.507113 0.489785 0.507233 0.505914 0.507039 0.493548 0.507009 0.496774 0.507085 0.506452 0.506867 0.493548 0.506943 0.503226 0.506871 0.505914 0.506694 0.491936 0.506804 0.505376 0.506651 0.497849 0.506605 0.49086 0.506695 0.50914 0.506493 0.491398 0.50659 0.503226 0.506452 0.498387 0.506388 0.49086 0.506498 0.50914 0.506329 0.49086 0.506421 0.505376 0.50631 0.495161 0.506282 0.494624 0.506343 0.506452 0.506172 0.492473 0.506247 0.497849 0.506228 0.505914 0.506107 0.490323 0.506186 0.505914 0.506089 0.501075 0.50602 0.489247 0.506104 0.509677 0.505972 0.493011 0.506029 0.499462 0.506022 0.507527 0.505902 0.490323 0.505982 0.503763 0.50589 0.496237 0.505854 0.493011 0.50592 0.508064 0.505802 0.491936 0.505861 0.503226 0.505833 0.502688 0.505748 0.488172 0.50584 0.506452 0.505735 0.495161 0.505753 0.494086 0.505777 0.506452 0.505664 0.492473 0.505731 0.502688 0.505685 0.502688 0.505603 0.490323 0.505684 0.504301 0.505588 0.496237 0.505581 0.494086 0.505635 0.508064 0.505527 0.491936 0.505576 0.5 0.505485 0.494086 0.505543 0.504301 0.505522 0.502151 0.505444 0.489785 0.505514 0.505914 0.505428 0.497849 0.505409 0.493011 0.505469 0.508602 0.505376 0.491936 0.505422 0.499462 0.505413 0.507527 0.505332 0.491398 0.505394 0.506452 0.505324 0.501075 0.505289 0.49086 0.505355 0.507527 0.505273 0.493548 0.505321 0.5 0.505241 0.495699 0.505301 0.503763 0.505276 0.503226 0.505215 0.49086 0.505282 0.506452 0.505201 0.494086 0.505219 0.497312 0.505236 0.506989 0.505153 0.492473 0.505206 0.503763 0.50518 0.498387 0.505143 0.490323 0.505191 0.508602 0.505119 0.494086 0.50514 0.498925 0.50513 0.501075 0.505064 0.491398 0.505126 0.506452 0.505061 0.495161 0.505056 0.494624 0.505098 0.508602 0.505028 0.491398 0.505073 0.499462 0.505065 0.504301 0.504995 0.489247 0.505058 0.505914 0.505 0.494624 0.50502 0.501075 0.504991 0.502151 0.504956 0.491936 0.505003 0.507527 0.504951 0.494086 0.50496 0.495161 0.504987 0.508064 0.504916 0.49086 0.504962 0.504839 0.504908 0.497849 0.504891 0.490323 0.504943 0.507527 0.504887 0.494086 0.504913 0.496774 0.504925 0.508064 0.504864 0.49086 0.504915 0.504839 0.50487 0.494086 0.50487 0.496774 0.504898 0.506452 0.504842 0.490323 0.504885 0.503763 0.50486 0.497849 0.504836 0.491936 0.504872 0.50914 0.504818 0.493548 0.504854 0.499462 0.504848 0.505376 0.5048 0.48871 0.50485 0.507527 0.504792 0.491398 0.50483 0.503226 0.504795 0.5 0.504827 0.501075 0.504787 0.496237 0.504782 0.491398 0.504822 0.508602 0.504763 0.493011 0.504794 0.498387 0.504797 0.506989 0.504755 0.49086 0.5048 0.505376 0.504745 0.491936 0.50477 0.502688 0.504757 0.502688 0.504714 0.489785 0.504758 0.505914 0.504716 0.497849 0.504714 0.491398 0.504751 0.507527 0.504704 0.493011 0.504728 0.497849 0.504732 0.507527 0.504685 0.49086 0.50473 0.504839 0.504692 0.494624 0.504695 0.495161 0.504719 0.506989 0.504674 0.49086 0.504705 0.504839 0.50468 0.496774 0.504669 0.492473 0.504699 0.506989 0.504658 0.492473 0.504683 0.497849 0.504694 0.506989 0.50465 0.487097 0.504693 0.504839 0.504657 0.498925 0.504646 0.492473 0.504678 0.506452 0.50464 0.493548 0.504674 0.501075 0.504673 0.505914 0.504638 0.49086 0.504679 0.505914 0.504635 0.494624 0.504645 0.493548 0.504672 0.507527 0.504626 0.491936 0.504659 0.501075 0.504649 0.505914 0.504613 0.490323 0.504655 0.505376 0.504626 0.499462 0.504614 0.49086 0.504648 0.508064 0.504609 0.494624 0.504643 0.498925 0.504635 0.505376 0.504596 0.490323 0.504635 0.506452 0.504607 0.500538 0.504604 0.491398 0.504635 0.508064 0.504599 0.493011 0.504621 0.498925 0.504627 0.508064 0.504585 0.491936 0.504628 0.503763 0.50461 0.496774 0.504594 0.491936 0.504624 0.508064 0.50459 0.490323 0.504618 0.501075 0.504621 0.504839 0.504582 0.492473 0.504616 0.503763 0.50459 0.496237 0.504593 0.494624 0.504617 0.507527 0.504575 0.493011 0.5046 0.503226 0.504577 0.500538 > EDIT: correct code Edited 2021-12-17 21:54 by CaptainBoing |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4044 |
It is mathematically almost bound to be so - the code divides any change by a big (& ever bigger) number. For many reasons one would commonly not want largely to ignore recent changes in that way. For "unbuffered open lines" - er, why? Not my idea of a remotely fair test! John |
||||
| CaptainBoing Guru Joined: 07/09/2016 Location: United KingdomPosts: 2170 |
and I alluded to such in my post. The idea was I wanted to see how varying "noisy" signals were smoothed by the algorithm... job done I'd say |
||||
| andreas Senior Member Joined: 07/12/2020 Location: GermanyPosts: 226 |
Hi John, averaging allways results in values which are not exact measurements, but they represent the system under test often better than a random error in a measuremnt. You decide over the "lag" by the Length parameter. It could be 1 or 2 ;-) if n = 1 then M = M + ( newvalue - M ) / 1 = M + newvalue - M = newvalue That means if n = 1 then the mean M is exactly the measured newvalue, which is correct. if n = 2 then M = M + ( newvalue - M ) / 2 = M + newvalue/2 - M/2 = M/2 + newvalue/2 = (M + newvalue)/2 I think that should be correct, too. Now one can show that this is correct for any n+1 if it is correct for n - but not me in a forum post (it's called an inductive proof ;-) The "Length" parameter decides how adaptive M follows the real measurements. You can take a small one i.e. 3 or 10 if you need more "accurate" values when quick reaction is needed. Or you take large values 1000, 10000, 100000,.. if you have many measurements but want to have a stable, less hysteric system. It depends on the situation i.e. temperature in a room is not changing quickly but a moon rocket needs quick reaction to side winds for navigation. I used that formula for nunchuck an took 50 as "Length" parameter. The result was a very smooth movement of the cursor, but quickly enough to move a robot arm. I will look for EWMA you mentioned - that sounds interesting! -andreas |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4044 |
EWMA is often used in industrial systems, automotive applications and so on in my experience. In effect it gives the oldest value the least weight and the newest the most, whilst at the same time not really trusting the newest is noise-free. It would make more sense for your program to use a constant 'n' rather than increasing it in the loop. (You refer to something like that in your recent post.) I think constant 'n' is EWMA (mathematically, it's usually written differently). John Edited 2021-12-18 03:53 by JohnS |
||||
| andreas Senior Member Joined: 07/12/2020 Location: GermanyPosts: 226 |
Hi John, I thought a while about that sentence and came to the conclusion, that I have to explain the method a little further. There are two phases, when collecting data, measurements, etc. The first phase is when you gather data to fill all the buffers or space you have reserved in your apparatus. In the beginning you have only a single value, later a second one and so on. After the initial phase you have filled all buffers and space for storage of values and you have to start to throw away information. This is phase 2. Old values have to be discarded and new values come into play. The algorithm on top respects these two phases. n is increasing step by step but only up to am maximum value "Length" and then n is constand = Length. The trick is the Min() function. The importand point is, that as log as n < Length, M is always the mean of all values entered during phase 1. It is the same as if you would store all these n < Length values and sum them up and divide them by n. No information is lost - you get the real average of all values entered so far (but without the need of storage!) When n reaches Length the Min() function guaranties that n will get no larger than Length. n keeps constant now! The real advantage of the method is that you only need two variables and one constant to calculate the mean value of a lot of measurements. You get "correct" values just from the beginning. If you would store the values first into an array, sum them up and divide them by the length of the array - you would have to wait until the array is filled before you get the first result. And you would have to store all these values into your valuable ram! -andreas Edited 2021-12-18 06:29 by andreas |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4044 |
EWMA doesn't use an array. It just uses the new reading and the old EWMA. Doesn't even need a Length or Min(). I think mathematically yours ends up the same but with the added Length and Min. John Edited 2021-12-18 06:55 by JohnS |
||||
| andreas Senior Member Joined: 07/12/2020 Location: GermanyPosts: 226 |
I modified the program a little to show the 'n' together with the 'M' The 'sliding window' has now a Length of 10: > list ' running mean - get better measurements Dim M As float = 0.0 ' running mean Dim n As integer = 0 ' counter [0..Max] Const Length = 10 ' up to 'Length' values are counting While true do n = Min(Length, n + 1) M = M + (Pin(temp) - M ) / n Print n,M Pause 1000 Loop > run 1 20.01453921 2 19.78041347 3 19.85845538 4 19.89747634 5 19.92088891 6 19.85845538 7 19.81386 8 19.78041347 9 19.80642744 10 19.82723861 10 19.84596867 10 19.86282573 10 19.87799708 10 19.89165129 10 19.90394008 10 19.91499999 10 19.92495392 Min() is no array - it is just the minimum function of MMBasic. -andreas Edited 2021-12-18 07:46 by andreas |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4044 |
I can't test this but something like: dim ewma as float const weight = 0.1 'fraction of current reading to use ewma = pin(temp) while true do ewma = (1 - weight) * ewma + pin(temp) * weight print ewma pause 1000 loop (add a loop counter if you wish, but it's not needed) John |
||||
| andreas Senior Member Joined: 07/12/2020 Location: GermanyPosts: 226 |
Hello John, your algo works! I would say it weights the "old mean" 90% and the new measurement 10% if you take 0.1 as weight. I programmed this into the pico and it shows good results: > list Dim ewma As float Const weight = 0.1 ewma = Pin(temp) While true do ewma = (1-weight)*ewma + Pin(temp) * weight Print ewma Pause 1000 Loop > run 19.03121109 19.0358936 19.04010787 19.04390071 19.04731426 19.00356131 19.0110088 19.01771154 19.02374401 19.02917324 19.03405954 19.03845721 19.04241511 18.99915207 19.00704049 18.96731492 18.97838705 I see a possible problem only, if the first measurement ewma = Pin(temp) should measure a complete wrong value by accident. I suppose then it would take some rounds to get rid of that value. If I put a Length = 10 into my method, it yields simmilar results: ' running mean - get better measurements Dim M As float = 0.0 ' running mean Dim n As integer = 0 ' counter [0..Max] Const Length = 10 ' up to 'Length' values are counting Print " n"," Pin(temp)"," running mean" Print "---------------------------------------" While true do t = Pin(temp) n = Min(Length, n + 1) M = M + (t - M ) / n Print n,t,M Pause 1000 Loop n Pin(temp) running mean --------------------------------------- 1 18.60978475 18.60978475 2 18.60978475 18.60978475 3 19.07803624 18.76586858 4 19.07803624 18.8439105 5 18.60978475 18.79708535 6 18.60978475 18.76586858 7 18.60978475 18.74357089 8 19.07803624 18.78537906 9 19.07803624 18.81789652 10 19.07803624 18.8439105 10 19.07803624 18.86732307 10 19.07803624 18.88839439 10 18.60978475 18.86053342 10 18.60978475 18.83545856 10 19.07803624 18.85971632 10 18.60978475 18.83472317 10 19.07803624 18.85905447 10 19.07803624 18.88095265 But now I see a new problem: Why are some measurements identical up to the last digits, then jumping by 0.4 degrees back and forth? Looks like some strange quantisation inside the temperature measurement Pin(temp)? -andreas Edited 2021-12-18 20:12 by andreas |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 10310 |
From the RP2040 errata Differential nonlinearity (acronym DNL) Effective number of bits (ENOB)  |
||||
| Solar Mike Guru Joined: 08/02/2015 Location: New ZealandPosts: 1163 |
>>Looks like some strange quantisation inside the temperature measurement Pin(temp)? Not helped by the ADC reference being a very noisy 3.3v switching supply; I have had to use a LM4040 shunt reference connected to the ADC Ref pin to get more stable results. Even better turn off the internal 3.3V switcher and use an external 3.3V linear regulator. Kind of defeats the use of a cheap CPU... Mike |
||||
| Page 1 of 3 |
|||||
 Print this page |
 |
The Back Shed's forum code is written, and hosted, in Australia. | © JAQ Software 2025 |
