SHG (spectroheliography) software v2 update available now

The place for all your solar software; capture, stacking, post processing, ancillaries etc. It's all up for discussion here!
Post Reply
thesmiths
Almost There...
Almost There...
Posts: 1053
Joined: Mon Jun 04, 2012 8:13 pm
Location: London, England
Has thanked: 42 times
Been thanked: 1485 times

SHG (spectroheliography) software v2 update available now

Post by thesmiths »

We had good feedback from our initial English language version of Valerie's Solex_ser_recon software. We uploaded today a new and improved version which takes into account some suggestions from users, as well as an improvement in the geometry correction. The new geometry correction is not only faster but also seems to be more robust (i.e. it fails less often).

As before, the software is available from our GitHub: https://github.com/thelondonsmiths/Solex_ser_recon_EN

The core reconstruction code is the same as before (and has now also been implemented in Valerie's French language version). She recently implemented an improved geometry correction that uses a short Python routine called lsq-ellipse that performs a least squares fitting of ellipses. We have taken this approach as well and believe the results are potentially more accurate and robust than before.

If you have downloaded the previous version, you may want to keep that around somewhere in case you do not like the new version. The UI remains very similar to before. See our previous post for details viewtopic.php?f=33&t=32654. Make sure to run the new Windows batch file (windows_setup) in order to install the additional Python libraries.

updated GUI
updated GUI
new GUI.JPG (37.67 KiB) Viewed 2546 times

The "Show graphics" box allows the "live view" of the raw reconstruction, the pop up of a geometry correction window, and a summary window of the image results. The "Show graphics" feature, in my view, has more pedagogical value than anything else. A user made the request to change the FITS file extension to .fits from .fit (it turns out even a .fts extension exists). All extensions are the same file format but having different extensions can be confusing so we settled on just using the .fits extension.

Another user suggested code that allows running the program from a command line (rather than via the GUI) and we have implemented it in this update. See the README file for details.

If "Show graphics" is enabled, then after the "live view" reconstruction (which I always find entertaining ), a geometry correction window will appear, such as the one below:

geometry correction
geometry correction
Figure_1c.png (94.89 KiB) Viewed 2546 times

This is quite a fun representation for how the "rugby ball" image from the reconstruction is circularised into something that looks like an image of the Sun. It works by first doing a 4x4 downsizing of the data, then applying a small gaussian blur, then doing edge detection. The exterior edge is then fitted to an ellipse, which provides the data to transform the image into a circle. The 4x4 downsizing turned out to provide a significant speed up in the geometry correction.

Finally, a window appears with a summary of solar images with different image processing applied. In my opinion, the CLAHE (Contrast Limited Adaptive Histogram Equalization) is generally the best one. A new feature, also suggested by a user, is to put a black disk on the protuberance image. This makes the image not only easier on the eye but also provides a visual check of how well the geometry correction function has achieved circularisation.

quad box summary
quad box summary
quad preview.JPG (164.86 KiB) Viewed 2546 times

If you need an SER file to test the software, I can suggest this one from Christian Buil: http://www.astrosurf.com/solex/download/16_33_15.zip
If you run this file (1.46 GB, 4461 frames, 2200x80 pixels) without "Show graphics", the new software will complete all the tasks of image reconstruction, automated geometry correction, defect removal and levels adjustment in around 5 seconds.


thesmiths
Almost There...
Almost There...
Posts: 1053
Joined: Mon Jun 04, 2012 8:13 pm
Location: London, England
Has thanked: 42 times
Been thanked: 1485 times

Re: SHG (spectroheliography) software v2 update available now

Post by thesmiths »

We just pushed an update that implements the writing of an SHG.ini file in the same directory as the program. This stores the location of the last source file used. Potentially, other "last used settings" or default settings could also be added here.

Also, code was added so the program will work under Linux, Mac, Windows 7 and Windows 10. This code was contributed by GitHub user Butet (djibb), who also contributed the code to implement the Command Line capabilities.


mconsidine
Oh, I get it now!
Oh, I get it now!
Posts: 22
Joined: Sun Aug 21, 2016 10:57 pm
Been thanked: 13 times

Re: SHG (spectroheliography) software v2 update available now

Post by mconsidine »

As noted in a post that I just made (erroneously) to the Spectroscopy forum, I have a version of this code which also supports AVI files. It may be found here for testing:

https://github.com/mattconsidine/Solex_ser_recon_AVI

There is also a tweak to the generation of the prom image which I have added (the code for the old version is there, but commented out).

@thesmiths: for reasons I do not understand I am not able to fork a copy of your repository like I can with others. That said, you are welcome to any of the changes you think would be useful - I'd submit a pull request if I could.

Hope this helps,
Matt

EDIT on Aug 2, 2021 to correct URL
Last edited by mconsidine on Mon Aug 02, 2021 3:28 pm, edited 1 time in total.


thesmiths
Almost There...
Almost There...
Posts: 1053
Joined: Mon Jun 04, 2012 8:13 pm
Location: London, England
Has thanked: 42 times
Been thanked: 1485 times

Re: SHG (spectroheliography) software v2 update available now

Post by thesmiths »

Hi Matt, I don't know why you can't make a fork. I see that someone else did (Butet). I'm not really a GitHub expert. Maybe you needed to create a fork before you made a pull request?


mconsidine
Oh, I get it now!
Oh, I get it now!
Posts: 22
Joined: Sun Aug 21, 2016 10:57 pm
Been thanked: 13 times

Re: SHG (spectroheliography) software v2 update available now

Post by mconsidine »

I've got to figure this out. When I try to fork other repositories, it seems to set up fine. When I try to look at yours the "Issues" button doesn't show up and when I hit "fork" I just end up with a version of Vdesnoux's. I'll keep poking at it ...
Matt


mconsidine
Oh, I get it now!
Oh, I get it now!
Posts: 22
Joined: Sun Aug 21, 2016 10:57 pm
Been thanked: 13 times

Re: SHG (spectroheliography) software v2 update available now

Post by mconsidine »

Okay, I figured this out. I think. For whatever reason, my prior forking of Vdesnoux's code prevented me from forking thelondonsmith's version. The solution? Create another github login.

So that has been done, the project forked, and I have submitted a commit reflecting the suggestions for AVI support. That code can be found here

https://github.com/mattconsidine/Solex_ser_recon_AVI
(It should be the same code base as found on the previously posted URL).

Now back to your regularly scheduled programming ...
mconsidine

EDITED (again) to reflect correct URL
Last edited by mconsidine on Mon Aug 02, 2021 3:29 pm, edited 2 times in total.


thesmiths
Almost There...
Almost There...
Posts: 1053
Joined: Mon Jun 04, 2012 8:13 pm
Location: London, England
Has thanked: 42 times
Been thanked: 1485 times

Re: SHG (spectroheliography) software v2 update available now

Post by thesmiths »

Hi Matt, I think the link in the post above is incorrect. It should be your other GitHub, I think.


mconsidine
Oh, I get it now!
Oh, I get it now!
Posts: 22
Joined: Sun Aug 21, 2016 10:57 pm
Been thanked: 13 times

Re: SHG (spectroheliography) software v2 update available now

Post by mconsidine »

Achh...

Apologies for that. The new link should be
https://github.com/mattconsidine/Solex_ser_recon_AVI

This is the branch to my fork of thesmiths version. Yes, the other github link should also have the same code and to work. But the one posted here will actually let you review/accept/deny the proposed commit.

Thanks for pointing that out.
mconsidine

EDIT on Aug 2, 2021 to correct URL
Last edited by mconsidine on Mon Aug 02, 2021 3:29 pm, edited 1 time in total.


mconsidine
Oh, I get it now!
Oh, I get it now!
Posts: 22
Joined: Sun Aug 21, 2016 10:57 pm
Been thanked: 13 times

Re: SHG (spectroheliography) software v2 update available now

Post by mconsidine »

Just an FYI to anyone else playing with this code. My very early, tentative, highly experimental effort at putting the video reading and the video displaying into separate threads suggests that there is a significant uptick in performance to be had by having a threaded version of this software. I suspect that taking a feed directly from a camera would be possible and allow for a more real-time exercise.

So that might be an area for further investigation if anyone is interested. I'll keep hacking away at my own effort in that area as well.

mconsidine


User avatar
Merlin66
Librarian
Librarian
Posts: 3970
Joined: Tue Dec 13, 2011 1:23 pm
Location: Junortoun, Australia
Has thanked: 173 times
Been thanked: 615 times
Contact:

Re: SHG (spectroheliography) software v2 update available now

Post by Merlin66 »

Matt,
Your AVI version appears to work well.
It has allowed the processing of 8bit AVI with the INTI software.
Thanks for all the hard work.


"Astronomical Spectroscopy - The Final Frontier" - to boldly go where few amateurs have gone before
https://groups.io/g/astronomicalspectroscopy  
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
thesmiths
Almost There...
Almost There...
Posts: 1053
Joined: Mon Jun 04, 2012 8:13 pm
Location: London, England
Has thanked: 42 times
Been thanked: 1485 times

Re: SHG (spectroheliography) software v2 update available now

Post by thesmiths »

We discovered an issue with the geometry correction that needs fixing so will likely come out with a new version that includes the geometry fix as well as the SER/AVI and 8/16bit features in the next week or so. I also have some old SHG data that I would not mind reprocessing.

A good way to add feature "toggles" without cluttering up the UI would be to make better use of the SHG.ini file. At the moment, the only entry is for the last used directory. But other display options, processing functions, and output options could be included there (rather like you can manually edit the firecapture.ini file). We might implement a few of these in the next version and then others could add additional configuration parameters. Valerie has told me she agrees that this is the best way to go since everyone seems to wants to customise the UI and to specify which data get displayed and what files get written.


User avatar
Merlin66
Librarian
Librarian
Posts: 3970
Joined: Tue Dec 13, 2011 1:23 pm
Location: Junortoun, Australia
Has thanked: 173 times
Been thanked: 615 times
Contact:

Re: SHG (spectroheliography) software v2 update available now

Post by Merlin66 »

Sounds good! Look forward to seeing a consolidated SER/AVI in English
The attached image has been extracted from an 8 bit AVI supplied by Fulvio Mete using INTI software V2 +Matt's AVI.
I use this AVI to provide a comparison between various processing methods in "Imaging Sunlight - using a digital spectroheliograph".
Wah's SLiM, Buil's IRIS, Defourneau's SpecHelio Bas, Paraskeva's BASS Project, Virtual Dub/ ImageJ and AstroSnap.

Ha0002_h_clahe.png
Ha0002_h_clahe.png (896.32 KiB) Viewed 2449 times


"Astronomical Spectroscopy - The Final Frontier" - to boldly go where few amateurs have gone before
https://groups.io/g/astronomicalspectroscopy  
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
User avatar
Merlin66
Librarian
Librarian
Posts: 3970
Joined: Tue Dec 13, 2011 1:23 pm
Location: Junortoun, Australia
Has thanked: 173 times
Been thanked: 615 times
Contact:

Re: SHG (spectroheliography) software v2 update available now

Post by Merlin66 »

The input screen from Matt's INTI AVI version.
Solex_AVI screen.png
Solex_AVI screen.png (76.27 KiB) Viewed 2448 times


"Astronomical Spectroscopy - The Final Frontier" - to boldly go where few amateurs have gone before
https://groups.io/g/astronomicalspectroscopy  
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
thesmiths
Almost There...
Almost There...
Posts: 1053
Joined: Mon Jun 04, 2012 8:13 pm
Location: London, England
Has thanked: 42 times
Been thanked: 1485 times

Re: SHG (spectroheliography) software v2 update available now

Post by thesmiths »

To Ken: so what is your opinion having compared all those different software? I only tried the Wah! approach, then tried the original Valerie software (both web based and what they can WINinti), before setting out to try improve on Valerie's. The Wah! methodology was very time consuming and I honestly could not get far with Christian's software. I guess the major parameters are quality of results, consistency of results, ease of use, and throughput.


User avatar
Merlin66
Librarian
Librarian
Posts: 3970
Joined: Tue Dec 13, 2011 1:23 pm
Location: Junortoun, Australia
Has thanked: 173 times
Been thanked: 615 times
Contact:

Re: SHG (spectroheliography) software v2 update available now

Post by Merlin66 »

Doug,
In terms of output I thought the SLiM and BASS Project were very good.
The ability to vary the pixel selection width and position as well as the tilt/ slant/ smile corrections was effective.
The reduction of Transversalium was marginal....
The final Y/X correction required further action....

Playing with Valerie's INTI was a bit of an eye opener. Compared to the previous software, it was FAST!!!
The automatic application of corrections and the Y/X correction gave great results.
Now it can handle both SER and AVI files I think it will become the "weapon of choice" for users of the digital SHG.
Thanks to all who have contributed to the software, their efforts have been acknowledged.


"Astronomical Spectroscopy - The Final Frontier" - to boldly go where few amateurs have gone before
https://groups.io/g/astronomicalspectroscopy  
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
thesmiths
Almost There...
Almost There...
Posts: 1053
Joined: Mon Jun 04, 2012 8:13 pm
Location: London, England
Has thanked: 42 times
Been thanked: 1485 times

Re: SHG (spectroheliography) software v2 update available now

Post by thesmiths »

I think the great concept of Valerie's approach was to attempt a fully automate process of reconstruction, transversalium removal and geometric distortion. Obviously, some compromises must be taken with the automated approach but the time saving is enormous. Our contribution has mainly been the increase in processing speed (around a 7x speed up) and the improvement in user interface (and translation into English, of course).

I also realise the important decision Valerie made to post her Python code on GitHub. This allowed us, and a few others, to move the code forward very rapidly. It is rather a "Cambrian explosion" at the moment. The struggle now is to keep all the various versions synchronised. Just as with biological diversity, there as separation into different "species" which can be very interesting but may cause issues down the road as it eventually becomes difficult to merge code.


Post Reply