Magnetometer sampling

I LOVE finding out about different ways to appreciate the Sun and light in general. Use this forum to post your info or questions about various outside the mainstream ways to appreciate our life giving star!
Post Reply
SimonM
Ohhhhhh My!
Ohhhhhh My!
Posts: 108
Joined: Mon Dec 28, 2020 1:01 am
Has thanked: 42 times
Been thanked: 49 times

Magnetometer sampling

Post by SimonM »

I have a couple of questions e.g. for Stu and also an update of where I reached with these interesting low cost sensors. For anyone contemplating trying this, it's easy to get something from the sensor, but much harder to get reliable data out. Other sensors e.g. integrated sensors (IC) are not sufficiently sensitive and only provide an indication of direction. The sensors from fgsensors.com are fairly unique products.

So the question is how often to sample e.g. once a minute, and for how long e.g. as long as possible, at least over 4, 8, 16 or 32 seconds?

The second question is, if the data was published about two locations e.g. London and Edinburgh, would it be meaningful to extrapolate how the field might be at a third (unmeasured) location e.g. Birmingham based on the difference and relative distances? I'm in Hampshire, but you get the idea.

I have previously experimented with a magnetometer from fgsensors.com using their fg-3+ sensor. They are sold individually, as a pair and also as a kit. I used their quick starter kit which provides a Arduino Shield with an LCD display a regulated PSU source and a divider. For E-W monitoring two sensors are not required so I arranged to purchase the kit with only one sensor. It isn't necessary to use the kit because the same function (without the LCD) can be built relatively easily and FWIW, the kit is no longer in production. Accept for the use of the LCD and being standalone, ease of h/w setup, that's not a huge disadvantage.

The sample Arduino code exists to show how to get data from a pair of sensors so that it can provide an indication of changes in flux. I simplified the code to only use one sensor. The operating principle used is quite simple: the digital pulses from the sensor can be counted over a timed interval and the frequencies of pulses can be displayed. The timed interval is setup to enable and disable an interrupt for the duration and the interrupts cause a counter to increment.

The data sheets for the sensor show that it is both voltage and temperature sensitive. Without allowances for this, the digital outputs cannot be relied upon because the variation in count frequency are small. If you are not careful, you will only get an indication of voltage and temperature variations because that the magnetometer output is swamped by other factors.

I used the Arduino to count the pulses over about half the sampling interval and the other half to monitor the temperature and regulate the heating of the sensor to maintain the temperature. I wasn't sure how often to sample the sensor and how often to apply changes to the heating to regulate the temperature. I used electric heating of an aluminium plate with a digital temperature chip and a glass vacuum flask to isolate the external environment.

Regulation of the voltage is a two stage process, the Arduino PSU is regulated to 12V and a further regulator is provided on the Arduino shield for only the sensor. Using a PID controller and FIR filter I could maintain a programmed temperature of the environment to within 1/16 C e.g. 38.0 C +0 C / - 0.0625 C indefinitely, regardless of the external temperature and collect/display the counts on the LCD. A PID controller is a software algorithm that takes the temperature difference (error) and calculates the changes required to heat (or cool) something e.g. my aluminium bar and maintain the temperature. The FIR filter is a software filter that averages the output and I used an 8 stage filter. One shortcoming of the Arduino UNO is that the frequency of operation varies with temperature and so the timing of pulses can't be relied upon and it was necessary to switch to a different model of Arduino to overcome this deficiency.

I programmed the Arduino to allows for a wide variation of sampling interval - from 1/4 second and up to a day. Also two types of sampling e.g. if the sampling was for every minute, it could sample for 1, 2, 4 or up to 32 seconds. Sampling over a longer time increases the accuracy but has the disadvantage if you want to do something else e.g. regulate the temperature, a PID circuit requires a more frequent temperature sampling to avoid time delays in making a change. To give an example using the second hand of a clock. Sampling could be for 32 seconds, starting at 12 O'Clock. equally, it could be for sixteen seconds at 12 O'Clock and again at 6 O 'Clock. Or eight seconds, four times a minute. It is still sampling once a minute and the total count would be over 32 seconds. However it is done, the cycle repeats: time, heat and wait. The wait is an important phase because electrical heating will create a temporary magnetic field e.g. the heating cannot be run continuously without also affecting the count and it can't be switched too infrequently to allow the bar to cool too much. Heating is proportional and uses only a small percentage of the available heating. The use of a vacuum flask makes the power consumption negligible.

The LCD was quite useful because I could arrange for initial readout after startup within two seconds and thereafter every minute. For alignment I found it better to keep the changing update frequency to a short interval e.g. to determine a maximum and minimum. The LCD provides a direct feedback and doesn't need a second computer to accept the "published" output and could show lots of other information like the temperature. The project is as much about heating, plumbing etc. as counting pulses. The Arduino is mostly idle so it can manage several tasks during the sampling interval, but only in distinct phases e.g. heat, time and delay. There are other ways to control heating and temperature regulation and its not necessary to keep the measurements in a digital only domain - just my approach to the problem.

I also have a Raspberry Pi which I intended to use as a web server, a database and controller of the Arduino. I started to change the Arduino code previously written to allow the sampling to be triggered by the Pi, but the timing of counts and temperature controlled by the Arduino. Software that collects data can be setup to use a push/pull model e.g. "scrape" data is a pull on request, but push is just as valid a way to do it and has the advantage that the Arduino can buffer the readings and need not always be connected. Trying to synchronise the operations in the Arduino and Pi proved (for me) just a bit too difficult, to "scrape" data. The Raspberry Pi is much more capable than an Arduino, but the simplicity of the Arduino means that you know exactly what is going on and this helps with timing and counting without additional hardware e.g. on a Pi only setup, the processor is not always available to do exactly what you want and counting accurately can't be relied upon.

Simon


User avatar
Carbon60
Way More Fun to Share It!!
Way More Fun to Share It!!
Posts: 14175
Joined: Wed Mar 07, 2012 12:33 pm
Location: Lancashire, UK
Has thanked: 8355 times
Been thanked: 8103 times

Re: Magnetometer sampling

Post by Carbon60 »

Hi Simon,

It sounds like you're having a lot of 'fun'. Temperature regulation, voltage regulation, frequency counting and data logging are tricky problems for sure. I found with much trial and error how to do these things whilst producing a satisfactory result. I started by using frequency counting, but found the results to be 'lumpy' with steps in the output chart because of the limitations in the cheap frequency counter I was using. I then used a flip-flop array to reduce the frequency from 60+KHz at the sensor to within the range of a sound card, but this was unreliable and again somewhat 'lumpy'. I ended up using a professionally made frequency down converter, as you will have seen in my write up.

The sensor is buried in my garden, protected from water ingress to somewhat keep the temperature fluctuations on a daily basis fairly modest, voltages are regulated using IC's and the frequency converter is housed in its own thermally controlled environment. Sampling is through my sound card and frequency is measured and logged using a dedicated PC running 24/7. A CSV file containing date/time/frequency data is created and off-line the data are charted using Excel.

To get to the core of your first question, my PC takes a reading of frequency every 150 seconds for my regular charts, but also I take readings every second for hi-res work. The actual sampling periods for the PC to make accurate frequency measurements are unknown to me (I guess it will say somewhere in Spectrum Lab). FYI, it is recorded to 3 decimal places in the 3-5KHz range.

The apparent magnetic field strength (East-West vector values) is calculated using the relationship between field and frequency. I usually present my data as 'rate of change' to avoid some of the imprints from the worst thermal effects, since for our purposes it is usually the rate of change in magnetic field that we're interested in. A 'loose' calibration can be achieved using a 'local' professional magnetometer operated by Intermagnet on the web.

This brings me to your second question about data extrapolation between observatories. In my experience the readings between relatively local observatories are close enough for our purposes. Don't expect your readings to correlate with Hawaii from the UK, or course, but observatories within the UK and closer regions of Europe will give similar results. Some details may vary, but mine here in the North West of England correlate very will with an Intermagnet observatory in France.

I hope this helps.

Do you have any data to share yet?

Stu.


H-alpha, WL and Ca II K imaging kit for various image scales.
Fluxgate Magnetometers (1s and 150s Cadence).
Radio meteor detector.
More images at http://www.flickr.com/photos/solarcarbon60/
SimonM
Ohhhhhh My!
Ohhhhhh My!
Posts: 108
Joined: Mon Dec 28, 2020 1:01 am
Has thanked: 42 times
Been thanked: 49 times

Re: Magnetometer sampling

Post by SimonM »

Hi Stu

Any frequency counter can be "lumpy" and so they provide an indication of frequency without repeatable accuracy. I found to get a repeatable accuracy e.g. 50,004 Hz and the same reading again and again, I had to count pulses over a longer time interval. The sample code measured the frequency over half a second which is too short an interval. A sound card may be sampling over a very small interval? So you have to average many samples to create one "super" sample?

Frequency counters work on one of two principles:

They either count the number of cycles in a given interval to determine the frequency - that is good for high frequency applications. It is usual to think of sampling for a number of seconds e.g. 1, 2, 4, 8, 16 seconds etc, and then dividing the output by the same amount. With binary division it's convenient to shift the output by one or more binary bits to effect a division by 2 or more. If you wanted to divide by 5, 10, 100 then the "cost" of division increases.

Or they time the length of one cycle - that is good for low frequencies. The issue with this is that the length of the cycle has to be measured from the same point in a sine wave. The good news is the output from the sensor is digital. By dividing the output by two or 4, the frequency is reduced to a manageable level and the output is buffered so that with a sharp rise time, the zero crossing isn't affected by noise too much and more accurate timings can be made. A sound card might time the length of one cycle, but it would be better would be to time it over multiple cycles. Some sound cards work like this.

Taking several short measurements and averaging them. isn't the same as taking measurements for a longer interval with either cycle or duration counting, but it helps. I used a FIR filter - to act as a low pass filter. Using the once per second example, it creates an output that is changing every second but also uses the last 8 data points. I wanted an instant result without waiting and that's possible with only one sample, then two, then four etc. That's why I could get a reading in two seconds (one second of counting after setup) and then get a smoothed output, until after 8 samples the result is smoothed. A FIR filter doesn't just average the result, it applies weights to each of the sample points and the total is usually a multiple of 1024 e.g. add all the (data x weights) and divide by 1024 to get a unity gain.

Burying the sensor in the garden is a good idea because it keeps the temperature fluctuations to a minimum but there will be wide variations between summer and winter and some variation during the day. There are published charts showing how wide or narrow the variation of temperature is e.g. at 10cm, 50cm, 1m or more. Although burying is a long term goal, I only managed to put the sensor (vacuum flask) out in the garden and ran a 5m cable back to a PC. There was a big difference between the sensor being outside and on a desk where things like a hard disk spinning up and down or the movement of a wrist with a watch make a difference.

Getting the temperature regulated was a lot of fun. A PID controller can be as simple as varying the varying the heating proportionally with the temperature error. Or the duty cycle of turning on/off proportional with the temperature error. That's a lot better than many circuits e.g. like a central heating circuit that switches the heating off when a preset is reached. Maintaining the temperature to within a tolerance is different to measuring it accurately... PID uses three constants for P, I and D. T get these values correct requires a lot of trial and error. P and I are proportional and integral and are sufficient but differential is interesting as it allow the output to stabilise quickly e.g. no overshoot. When it works it is like a car that quickly accelerates to 70 but doesn't overshoot. It's easy to get to 70 or easy to slowly get to 70 but to do it quickly requires some thought. When we drive we automatically do it, but getting a computer to do it isn't that easy.

The Arduino I use is a genuine Arduino. These have a better quality crystal, voltage regulation. The original Arduino used a crystal for the USB interface and a ceramic resonator for the processor. There are two processors. For timing, the use of a ceramic resonator isn't very stable. The newer version of the Arduino use only one crystal and no resonator so are much more accurate. They also provide the same number of input/outputs but use more pins so they are more flexible. They also have more timers and analog inputs. The best bit, they are cheaper...

Thanks for the answer to the first question. 60 seconds or 150 seconds were my preferred sample interval. If I use your 150 seconds as the sample interval, I can easily buffer up to a days worth (1440) of samples but still have enough for a day's graph. or me, that translates to eight samples of 8 seconds. The total count is added (reset) after 150 seconds and represents a sample interval open window of 64 seconds, spaced equally around a 150 second cycle. It is still a good idea to filter the data (a bit). Some of the remaining time is used to control the heating cycle and some time to wait when it's neither timing nor heating.

The answer to the second question is quite interesting because if most of the UK data is similar, it means that extracting the data over a longish period like 24 hours allows the equipment to be calibrated against a standard nearby that you have faith in. That's easier than trying to calibrate using a coil.

I'm also using a divider on the Arduino Shield. There are two four bit counters and I''m only using one of them. They divide the input by four using the second output and the Arduino triggers an interrupt on both edges. This means that the count needs to be divided by twice the sample time. Using my 8 x 8 seconds, that's dividing the count by 64 * 2 = 128, or 7 bit shifts. Dividing by 128 and rounding up/down is add 127 and shift 7 bits, which computers find easy. The Arduino is therefore not doing too much and has spare time for PID and FIR calculations or checking for requests e.g. to empty a buffer and send it to a second computer...

Collecting the data? I have either concentrated on sampling the data outside without controlling the temperature, or inside without having a stable environment that doesn't have interference e.g. metal that changes the result. Although the Arduino works collecting the data and turning it into something that is as simple as a count every sample interval, that's still a lot of data to store and I also kept the raw data and metrics about heating and other info. So no meaningful data to share. The Arduino isn't worked to capacity - most of the time it just waits doing nothing...

One interesting idea I had was directly using the data to enter automatically into a spreadsheet or database using keystrokes. The Arduino I have has an interesting feature that the output on the USB is also valid and seen by a PC or Mac as either a mouse or keyboard. It's called a HID device. Using Excel as an example, the setup can be left for a while to automatically send keystrokes to an open spreadsheet. The data can then be graphed but it's not the same as using a database, something I don't know too much about.

As well as drawing a graph, do you have a way to extract the data e.g. as a csv file? I could extract a days data to a file easier than graphing or storing it.

Simon


User avatar
Carbon60
Way More Fun to Share It!!
Way More Fun to Share It!!
Posts: 14175
Joined: Wed Mar 07, 2012 12:33 pm
Location: Lancashire, UK
Has thanked: 8355 times
Been thanked: 8103 times

Re: Magnetometer sampling

Post by Carbon60 »

Hi Simon,

I save the data as a CSV text file, logging date, time and frequency. I then transfer this on a USB stick from my dedicated PC to my laptop and load it into excel for further mathematical manipulation, filtering of any spikes and charting.

Stu.


H-alpha, WL and Ca II K imaging kit for various image scales.
Fluxgate Magnetometers (1s and 150s Cadence).
Radio meteor detector.
More images at http://www.flickr.com/photos/solarcarbon60/
Post Reply