SHG (spectroheliography) software v2 update available now
-
- Almost There...
- Posts: 1196
- Joined: Mon Jun 04, 2012 8:13 pm
- Location: London, England
- Has thanked: 119 times
- Been thanked: 1671 times
- Contact:
SHG (spectroheliography) software v2 update available now
We had good feedback from our initial English language version of Valerie's Solex_ser_recon software. We uploaded today a new and improved version which takes into account some suggestions from users, as well as an improvement in the geometry correction. The new geometry correction is not only faster but also seems to be more robust (i.e. it fails less often).
As before, the software is available from our GitHub: https://github.com/thelondonsmiths/Solex_ser_recon_EN
The core reconstruction code is the same as before (and has now also been implemented in Valerie's French language version). She recently implemented an improved geometry correction that uses a short Python routine called lsq-ellipse that performs a least squares fitting of ellipses. We have taken this approach as well and believe the results are potentially more accurate and robust than before.
If you have downloaded the previous version, you may want to keep that around somewhere in case you do not like the new version. The UI remains very similar to before. See our previous post for details viewtopic.php?f=33&t=32654. Make sure to run the new Windows batch file (windows_setup) in order to install the additional Python libraries.
The "Show graphics" box allows the "live view" of the raw reconstruction, the pop up of a geometry correction window, and a summary window of the image results. The "Show graphics" feature, in my view, has more pedagogical value than anything else. A user made the request to change the FITS file extension to .fits from .fit (it turns out even a .fts extension exists). All extensions are the same file format but having different extensions can be confusing so we settled on just using the .fits extension.
Another user suggested code that allows running the program from a command line (rather than via the GUI) and we have implemented it in this update. See the README file for details.
If "Show graphics" is enabled, then after the "live view" reconstruction (which I always find entertaining ), a geometry correction window will appear, such as the one below:
This is quite a fun representation for how the "rugby ball" image from the reconstruction is circularised into something that looks like an image of the Sun. It works by first doing a 4x4 downsizing of the data, then applying a small gaussian blur, then doing edge detection. The exterior edge is then fitted to an ellipse, which provides the data to transform the image into a circle. The 4x4 downsizing turned out to provide a significant speed up in the geometry correction.
Finally, a window appears with a summary of solar images with different image processing applied. In my opinion, the CLAHE (Contrast Limited Adaptive Histogram Equalization) is generally the best one. A new feature, also suggested by a user, is to put a black disk on the protuberance image. This makes the image not only easier on the eye but also provides a visual check of how well the geometry correction function has achieved circularisation.
If you need an SER file to test the software, I can suggest this one from Christian Buil: http://www.astrosurf.com/solex/download/16_33_15.zip
If you run this file (1.46 GB, 4461 frames, 2200x80 pixels) without "Show graphics", the new software will complete all the tasks of image reconstruction, automated geometry correction, defect removal and levels adjustment in around 5 seconds.
As before, the software is available from our GitHub: https://github.com/thelondonsmiths/Solex_ser_recon_EN
The core reconstruction code is the same as before (and has now also been implemented in Valerie's French language version). She recently implemented an improved geometry correction that uses a short Python routine called lsq-ellipse that performs a least squares fitting of ellipses. We have taken this approach as well and believe the results are potentially more accurate and robust than before.
If you have downloaded the previous version, you may want to keep that around somewhere in case you do not like the new version. The UI remains very similar to before. See our previous post for details viewtopic.php?f=33&t=32654. Make sure to run the new Windows batch file (windows_setup) in order to install the additional Python libraries.
The "Show graphics" box allows the "live view" of the raw reconstruction, the pop up of a geometry correction window, and a summary window of the image results. The "Show graphics" feature, in my view, has more pedagogical value than anything else. A user made the request to change the FITS file extension to .fits from .fit (it turns out even a .fts extension exists). All extensions are the same file format but having different extensions can be confusing so we settled on just using the .fits extension.
Another user suggested code that allows running the program from a command line (rather than via the GUI) and we have implemented it in this update. See the README file for details.
If "Show graphics" is enabled, then after the "live view" reconstruction (which I always find entertaining ), a geometry correction window will appear, such as the one below:
This is quite a fun representation for how the "rugby ball" image from the reconstruction is circularised into something that looks like an image of the Sun. It works by first doing a 4x4 downsizing of the data, then applying a small gaussian blur, then doing edge detection. The exterior edge is then fitted to an ellipse, which provides the data to transform the image into a circle. The 4x4 downsizing turned out to provide a significant speed up in the geometry correction.
Finally, a window appears with a summary of solar images with different image processing applied. In my opinion, the CLAHE (Contrast Limited Adaptive Histogram Equalization) is generally the best one. A new feature, also suggested by a user, is to put a black disk on the protuberance image. This makes the image not only easier on the eye but also provides a visual check of how well the geometry correction function has achieved circularisation.
If you need an SER file to test the software, I can suggest this one from Christian Buil: http://www.astrosurf.com/solex/download/16_33_15.zip
If you run this file (1.46 GB, 4461 frames, 2200x80 pixels) without "Show graphics", the new software will complete all the tasks of image reconstruction, automated geometry correction, defect removal and levels adjustment in around 5 seconds.
-
- Almost There...
- Posts: 1196
- Joined: Mon Jun 04, 2012 8:13 pm
- Location: London, England
- Has thanked: 119 times
- Been thanked: 1671 times
- Contact:
Re: SHG (spectroheliography) software v2 update available now
We just pushed an update that implements the writing of an SHG.ini file in the same directory as the program. This stores the location of the last source file used. Potentially, other "last used settings" or default settings could also be added here.
Also, code was added so the program will work under Linux, Mac, Windows 7 and Windows 10. This code was contributed by GitHub user Butet (djibb), who also contributed the code to implement the Command Line capabilities.
Also, code was added so the program will work under Linux, Mac, Windows 7 and Windows 10. This code was contributed by GitHub user Butet (djibb), who also contributed the code to implement the Command Line capabilities.
-
- Oh, I get it now!
- Posts: 22
- Joined: Sun Aug 21, 2016 10:57 pm
- Been thanked: 13 times
Re: SHG (spectroheliography) software v2 update available now
As noted in a post that I just made (erroneously) to the Spectroscopy forum, I have a version of this code which also supports AVI files. It may be found here for testing:
https://github.com/mattconsidine/Solex_ser_recon_AVI
There is also a tweak to the generation of the prom image which I have added (the code for the old version is there, but commented out).
@thesmiths: for reasons I do not understand I am not able to fork a copy of your repository like I can with others. That said, you are welcome to any of the changes you think would be useful - I'd submit a pull request if I could.
Hope this helps,
Matt
EDIT on Aug 2, 2021 to correct URL
https://github.com/mattconsidine/Solex_ser_recon_AVI
There is also a tweak to the generation of the prom image which I have added (the code for the old version is there, but commented out).
@thesmiths: for reasons I do not understand I am not able to fork a copy of your repository like I can with others. That said, you are welcome to any of the changes you think would be useful - I'd submit a pull request if I could.
Hope this helps,
Matt
EDIT on Aug 2, 2021 to correct URL
Last edited by mconsidine on Mon Aug 02, 2021 3:28 pm, edited 1 time in total.
-
- Almost There...
- Posts: 1196
- Joined: Mon Jun 04, 2012 8:13 pm
- Location: London, England
- Has thanked: 119 times
- Been thanked: 1671 times
- Contact:
Re: SHG (spectroheliography) software v2 update available now
Hi Matt, I don't know why you can't make a fork. I see that someone else did (Butet). I'm not really a GitHub expert. Maybe you needed to create a fork before you made a pull request?
-
- Oh, I get it now!
- Posts: 22
- Joined: Sun Aug 21, 2016 10:57 pm
- Been thanked: 13 times
Re: SHG (spectroheliography) software v2 update available now
I've got to figure this out. When I try to fork other repositories, it seems to set up fine. When I try to look at yours the "Issues" button doesn't show up and when I hit "fork" I just end up with a version of Vdesnoux's. I'll keep poking at it ...
Matt
Matt
-
- Oh, I get it now!
- Posts: 22
- Joined: Sun Aug 21, 2016 10:57 pm
- Been thanked: 13 times
Re: SHG (spectroheliography) software v2 update available now
Okay, I figured this out. I think. For whatever reason, my prior forking of Vdesnoux's code prevented me from forking thelondonsmith's version. The solution? Create another github login.
So that has been done, the project forked, and I have submitted a commit reflecting the suggestions for AVI support. That code can be found here
https://github.com/mattconsidine/Solex_ser_recon_AVI
(It should be the same code base as found on the previously posted URL).
Now back to your regularly scheduled programming ...
mconsidine
EDITED (again) to reflect correct URL
So that has been done, the project forked, and I have submitted a commit reflecting the suggestions for AVI support. That code can be found here
https://github.com/mattconsidine/Solex_ser_recon_AVI
(It should be the same code base as found on the previously posted URL).
Now back to your regularly scheduled programming ...
mconsidine
EDITED (again) to reflect correct URL
Last edited by mconsidine on Mon Aug 02, 2021 3:29 pm, edited 2 times in total.
-
- Almost There...
- Posts: 1196
- Joined: Mon Jun 04, 2012 8:13 pm
- Location: London, England
- Has thanked: 119 times
- Been thanked: 1671 times
- Contact:
Re: SHG (spectroheliography) software v2 update available now
Hi Matt, I think the link in the post above is incorrect. It should be your other GitHub, I think.
-
- Oh, I get it now!
- Posts: 22
- Joined: Sun Aug 21, 2016 10:57 pm
- Been thanked: 13 times
Re: SHG (spectroheliography) software v2 update available now
Achh...
Apologies for that. The new link should be
https://github.com/mattconsidine/Solex_ser_recon_AVI
This is the branch to my fork of thesmiths version. Yes, the other github link should also have the same code and to work. But the one posted here will actually let you review/accept/deny the proposed commit.
Thanks for pointing that out.
mconsidine
EDIT on Aug 2, 2021 to correct URL
Apologies for that. The new link should be
https://github.com/mattconsidine/Solex_ser_recon_AVI
This is the branch to my fork of thesmiths version. Yes, the other github link should also have the same code and to work. But the one posted here will actually let you review/accept/deny the proposed commit.
Thanks for pointing that out.
mconsidine
EDIT on Aug 2, 2021 to correct URL
Last edited by mconsidine on Mon Aug 02, 2021 3:29 pm, edited 1 time in total.
-
- Oh, I get it now!
- Posts: 22
- Joined: Sun Aug 21, 2016 10:57 pm
- Been thanked: 13 times
Re: SHG (spectroheliography) software v2 update available now
Just an FYI to anyone else playing with this code. My very early, tentative, highly experimental effort at putting the video reading and the video displaying into separate threads suggests that there is a significant uptick in performance to be had by having a threaded version of this software. I suspect that taking a feed directly from a camera would be possible and allow for a more real-time exercise.
So that might be an area for further investigation if anyone is interested. I'll keep hacking away at my own effort in that area as well.
mconsidine
So that might be an area for further investigation if anyone is interested. I'll keep hacking away at my own effort in that area as well.
mconsidine
- Merlin66
- Librarian
- Posts: 3973
- Joined: Tue Dec 13, 2011 1:23 pm
- Location: Junortoun, Australia
- Has thanked: 174 times
- Been thanked: 622 times
- Contact:
Re: SHG (spectroheliography) software v2 update available now
Matt,
Your AVI version appears to work well.
It has allowed the processing of 8bit AVI with the INTI software.
Thanks for all the hard work.
Your AVI version appears to work well.
It has allowed the processing of 8bit AVI with the INTI software.
Thanks for all the hard work.
"Astronomical Spectroscopy - The Final Frontier" - to boldly go where few amateurs have gone before
https://groups.io/g/astronomicalspectroscopy
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
https://groups.io/g/astronomicalspectroscopy
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
-
- Almost There...
- Posts: 1196
- Joined: Mon Jun 04, 2012 8:13 pm
- Location: London, England
- Has thanked: 119 times
- Been thanked: 1671 times
- Contact:
Re: SHG (spectroheliography) software v2 update available now
We discovered an issue with the geometry correction that needs fixing so will likely come out with a new version that includes the geometry fix as well as the SER/AVI and 8/16bit features in the next week or so. I also have some old SHG data that I would not mind reprocessing.
A good way to add feature "toggles" without cluttering up the UI would be to make better use of the SHG.ini file. At the moment, the only entry is for the last used directory. But other display options, processing functions, and output options could be included there (rather like you can manually edit the firecapture.ini file). We might implement a few of these in the next version and then others could add additional configuration parameters. Valerie has told me she agrees that this is the best way to go since everyone seems to wants to customise the UI and to specify which data get displayed and what files get written.
A good way to add feature "toggles" without cluttering up the UI would be to make better use of the SHG.ini file. At the moment, the only entry is for the last used directory. But other display options, processing functions, and output options could be included there (rather like you can manually edit the firecapture.ini file). We might implement a few of these in the next version and then others could add additional configuration parameters. Valerie has told me she agrees that this is the best way to go since everyone seems to wants to customise the UI and to specify which data get displayed and what files get written.
- Merlin66
- Librarian
- Posts: 3973
- Joined: Tue Dec 13, 2011 1:23 pm
- Location: Junortoun, Australia
- Has thanked: 174 times
- Been thanked: 622 times
- Contact:
Re: SHG (spectroheliography) software v2 update available now
Sounds good! Look forward to seeing a consolidated SER/AVI in English
The attached image has been extracted from an 8 bit AVI supplied by Fulvio Mete using INTI software V2 +Matt's AVI.
I use this AVI to provide a comparison between various processing methods in "Imaging Sunlight - using a digital spectroheliograph".
Wah's SLiM, Buil's IRIS, Defourneau's SpecHelio Bas, Paraskeva's BASS Project, Virtual Dub/ ImageJ and AstroSnap.
The attached image has been extracted from an 8 bit AVI supplied by Fulvio Mete using INTI software V2 +Matt's AVI.
I use this AVI to provide a comparison between various processing methods in "Imaging Sunlight - using a digital spectroheliograph".
Wah's SLiM, Buil's IRIS, Defourneau's SpecHelio Bas, Paraskeva's BASS Project, Virtual Dub/ ImageJ and AstroSnap.
"Astronomical Spectroscopy - The Final Frontier" - to boldly go where few amateurs have gone before
https://groups.io/g/astronomicalspectroscopy
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
https://groups.io/g/astronomicalspectroscopy
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
- Merlin66
- Librarian
- Posts: 3973
- Joined: Tue Dec 13, 2011 1:23 pm
- Location: Junortoun, Australia
- Has thanked: 174 times
- Been thanked: 622 times
- Contact:
Re: SHG (spectroheliography) software v2 update available now
The input screen from Matt's INTI AVI version.
"Astronomical Spectroscopy - The Final Frontier" - to boldly go where few amateurs have gone before
https://groups.io/g/astronomicalspectroscopy
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
https://groups.io/g/astronomicalspectroscopy
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
-
- Almost There...
- Posts: 1196
- Joined: Mon Jun 04, 2012 8:13 pm
- Location: London, England
- Has thanked: 119 times
- Been thanked: 1671 times
- Contact:
Re: SHG (spectroheliography) software v2 update available now
To Ken: so what is your opinion having compared all those different software? I only tried the Wah! approach, then tried the original Valerie software (both web based and what they can WINinti), before setting out to try improve on Valerie's. The Wah! methodology was very time consuming and I honestly could not get far with Christian's software. I guess the major parameters are quality of results, consistency of results, ease of use, and throughput.
- Merlin66
- Librarian
- Posts: 3973
- Joined: Tue Dec 13, 2011 1:23 pm
- Location: Junortoun, Australia
- Has thanked: 174 times
- Been thanked: 622 times
- Contact:
Re: SHG (spectroheliography) software v2 update available now
Doug,
In terms of output I thought the SLiM and BASS Project were very good.
The ability to vary the pixel selection width and position as well as the tilt/ slant/ smile corrections was effective.
The reduction of Transversalium was marginal....
The final Y/X correction required further action....
Playing with Valerie's INTI was a bit of an eye opener. Compared to the previous software, it was FAST!!!
The automatic application of corrections and the Y/X correction gave great results.
Now it can handle both SER and AVI files I think it will become the "weapon of choice" for users of the digital SHG.
Thanks to all who have contributed to the software, their efforts have been acknowledged.
In terms of output I thought the SLiM and BASS Project were very good.
The ability to vary the pixel selection width and position as well as the tilt/ slant/ smile corrections was effective.
The reduction of Transversalium was marginal....
The final Y/X correction required further action....
Playing with Valerie's INTI was a bit of an eye opener. Compared to the previous software, it was FAST!!!
The automatic application of corrections and the Y/X correction gave great results.
Now it can handle both SER and AVI files I think it will become the "weapon of choice" for users of the digital SHG.
Thanks to all who have contributed to the software, their efforts have been acknowledged.
"Astronomical Spectroscopy - The Final Frontier" - to boldly go where few amateurs have gone before
https://groups.io/g/astronomicalspectroscopy
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
https://groups.io/g/astronomicalspectroscopy
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
-
- Almost There...
- Posts: 1196
- Joined: Mon Jun 04, 2012 8:13 pm
- Location: London, England
- Has thanked: 119 times
- Been thanked: 1671 times
- Contact:
Re: SHG (spectroheliography) software v2 update available now
I think the great concept of Valerie's approach was to attempt a fully automate process of reconstruction, transversalium removal and geometric distortion. Obviously, some compromises must be taken with the automated approach but the time saving is enormous. Our contribution has mainly been the increase in processing speed (around a 7x speed up) and the improvement in user interface (and translation into English, of course).
I also realise the important decision Valerie made to post her Python code on GitHub. This allowed us, and a few others, to move the code forward very rapidly. It is rather a "Cambrian explosion" at the moment. The struggle now is to keep all the various versions synchronised. Just as with biological diversity, there as separation into different "species" which can be very interesting but may cause issues down the road as it eventually becomes difficult to merge code.
I also realise the important decision Valerie made to post her Python code on GitHub. This allowed us, and a few others, to move the code forward very rapidly. It is rather a "Cambrian explosion" at the moment. The struggle now is to keep all the various versions synchronised. Just as with biological diversity, there as separation into different "species" which can be very interesting but may cause issues down the road as it eventually becomes difficult to merge code.