Hi All.
I use AS3 for stacking my images, but I find it can be very time consuming when used in isolation, especially when there is 100's Gb of data to go through. Not sure if other people are using PIPP prior to AS3 to speed up post processing?
If I take a 4000 frame avi, for example, I run it through PIPP to select the best frames, normally 250. PIPP saves this as a 250 frame avi which I then run through AS3 and stack as I would normally.
PIPP seems happy to run on the fly while i'm capturing in Firecapture, and doesn't make FC to drop the frame rate, so, I may take a series of say 10 x 4000 frame avis of a certain target, and when I switch to the next target / wavelength / scope I get PIPP to turn this into 10 x 250 frame avis while I go on capturing more frames in FC. PIPP also has a 'join' feature so in the case of the 10 x 4000 frame avi it will treat this as 1 x 40000 frame avi and select the best (for example) 250 frame avi, which can sometimes work better.
If, like me, you find there sometimes just aren't enough hours in the day to get your processing done, you will find PIPP gives you some of this time back.
Hope is of use to people.
Mark
PIPP To Speed Up Image Processing Time
- marktownley
- Librarian
- Posts: 42280
- Joined: Tue Oct 18, 2011 5:27 pm
- Location: Brierley Hills, UK
- Has thanked: 20448 times
- Been thanked: 10254 times
- Contact:
PIPP To Speed Up Image Processing Time
http://brierleyhillsolar.blogspot.co.uk/
Solar images, a collection of all the most up to date live solar data on the web, imaging & processing tutorials - please take a look!
- rsfoto
- Way More Fun to Share It!!
- Posts: 6171
- Joined: Mon Jun 18, 2012 8:30 pm
- Location: San Luis Potosi, México
- Has thanked: 9431 times
- Been thanked: 5588 times
Re: PIPP To Speed Up Image Processing Time
Hi Mark,
Interesting. Do you really need 4000 frames in each AVI ?
Interesting. Do you really need 4000 frames in each AVI ?
regards Rainer
Observatorio Real de 14
San Luis Potosi Mexico
North 22° West 101°
Observatorio Real de 14
San Luis Potosi Mexico
North 22° West 101°
- MalVeauX
- Way More Fun to Share It!!
- Posts: 1858
- Joined: Tue May 09, 2017 7:58 pm
- Location: Florida
- Has thanked: 1171 times
- Been thanked: 1360 times
Re: PIPP To Speed Up Image Processing Time
Hi Mark,
I do pre-processing in PIPP (only quality weighting, no other processing) of my large piles of data. I reduce them down to 200~400 frames tops, depending on what I'm doing, because I generate quite often single files of 3.4k to 10.1k frames each go with the IMX290 sensor at 170 FPS or more.
I tested it quite a bit on several platforms, pre-processing data in PIPP, AS!3 and a PSS, on older architecture, newer architecture, intel vs AMD, etc, with some collaboration and it shows how inefficient all the software really is for multi-core use. AS!3 in particular struggles to handle large files like that from the 20Mp IMX183 sensor. I showed on various platforms that using PIPP first to quality sort the bulk of the frames and toss the worst ones shaved off significant amounts of time. And that AS!3 performed faster on smaller data sets. And that the overall final time savings was significant on my end, saving me literally hours per week in terms of copying time and pre-processing time of my data sets when I image. I compared the outputs of PIPP+AS!3 to just AS!3 and also PSS and the images layered over each other and subtracted showed so little difference that I was confident in saying that there's no outcome difference to speak of. So you have nothing but time savings to gain by using the software in tandem.
I have data, times, screen shots, CPU activity, etc, all here in this thread:
https://www.cloudynights.com/topic/7011 ... technical/
Capturing video and generating data with your CMOS camera also has virtually 0 CPU use. So you can use PIPP and pre-process data while capturing more data. I do this to save a lot of time.
Very best,
I do pre-processing in PIPP (only quality weighting, no other processing) of my large piles of data. I reduce them down to 200~400 frames tops, depending on what I'm doing, because I generate quite often single files of 3.4k to 10.1k frames each go with the IMX290 sensor at 170 FPS or more.
I tested it quite a bit on several platforms, pre-processing data in PIPP, AS!3 and a PSS, on older architecture, newer architecture, intel vs AMD, etc, with some collaboration and it shows how inefficient all the software really is for multi-core use. AS!3 in particular struggles to handle large files like that from the 20Mp IMX183 sensor. I showed on various platforms that using PIPP first to quality sort the bulk of the frames and toss the worst ones shaved off significant amounts of time. And that AS!3 performed faster on smaller data sets. And that the overall final time savings was significant on my end, saving me literally hours per week in terms of copying time and pre-processing time of my data sets when I image. I compared the outputs of PIPP+AS!3 to just AS!3 and also PSS and the images layered over each other and subtracted showed so little difference that I was confident in saying that there's no outcome difference to speak of. So you have nothing but time savings to gain by using the software in tandem.
I have data, times, screen shots, CPU activity, etc, all here in this thread:
https://www.cloudynights.com/topic/7011 ... technical/
Capturing video and generating data with your CMOS camera also has virtually 0 CPU use. So you can use PIPP and pre-process data while capturing more data. I do this to save a lot of time.
Very best,
- marktownley
- Librarian
- Posts: 42280
- Joined: Tue Oct 18, 2011 5:27 pm
- Location: Brierley Hills, UK
- Has thanked: 20448 times
- Been thanked: 10254 times
- Contact:
Re: PIPP To Speed Up Image Processing Time
Thanks Marty, that link is very informative, for reference the (basic) laptop i'm using is a i7 2.4Ghz 8Gb 256Gb. PC specs aside, my key takeaway here is that PIPP sorts frames using a global quality estimation (based on a single 'whole' frame'), AS3 does it's quality estimation based on 'local' alignment points.
This makes sense when I reflect on the resultant quality graph in AS3 as it can be quite 'up and down'. I'm saving 500 frames from PIPP but only stacking 96/128/256 in AS3 (depending on the quality of data), so, maybe there is some mileage in giving AS3 more frames to play with than used in the final stack size?
I'm capturing 4000 frames Rainer when i'm fighting the limit of seeing with the maximum aperture - within that 4000 there will be a hundred or so good ones that I can stack. If i'm using an aperture that is less than that dictated by seeing (eg Ha full disk) then I don't capture anywhere near that many. 4000 frame captures are only for the realms of hi-res work.
This makes sense when I reflect on the resultant quality graph in AS3 as it can be quite 'up and down'. I'm saving 500 frames from PIPP but only stacking 96/128/256 in AS3 (depending on the quality of data), so, maybe there is some mileage in giving AS3 more frames to play with than used in the final stack size?
I'm capturing 4000 frames Rainer when i'm fighting the limit of seeing with the maximum aperture - within that 4000 there will be a hundred or so good ones that I can stack. If i'm using an aperture that is less than that dictated by seeing (eg Ha full disk) then I don't capture anywhere near that many. 4000 frame captures are only for the realms of hi-res work.
http://brierleyhillsolar.blogspot.co.uk/
Solar images, a collection of all the most up to date live solar data on the web, imaging & processing tutorials - please take a look!
- p_zetner
- Way More Fun to Share It!!
- Posts: 1644
- Joined: Sun Mar 18, 2012 4:59 pm
- Location: Winnipeg, Manitoba, Canada
- Has thanked: 166 times
- Been thanked: 719 times
Re: PIPP To Speed Up Image Processing Time
Thanks Mark and Marty.
This sounds like a great timesaver and I will definitely experiment with it.
Cheers.
Peter
This sounds like a great timesaver and I will definitely experiment with it.
Cheers.
Peter
- MalVeauX
- Way More Fun to Share It!!
- Posts: 1858
- Joined: Tue May 09, 2017 7:58 pm
- Location: Florida
- Has thanked: 1171 times
- Been thanked: 1360 times
Re: PIPP To Speed Up Image Processing Time
Indeed, global vs local makes a big difference both in results and in CPU cycles needed to compute. PIPP is faster for that reason as you pointed out, AS!3 can of course be told to do global or local (there's zero benefit to using global frankly, especially in the context of solar imaging). Knowing this, I tested the outcomes to see if it would matter. And in that thread you can see the results, with the two outcomes layered over each other and subtracted. There's virtually no difference. So while PIPP is using global, it makes sense that the frames with the highest overall contrast in the entire frame would likely also have the best contrast in different areas within those frames as well. So using that logic, I use PIPP to quality sort pre-process the bulk of my thousands of frames and give me the best contrast few hundred frames. Then let AS!3 use local AP method (which is a mosaic method, if anyone was wondering, the way AP works is the size of the AP point is the area that will be examined, per frame, and the best from each frame in that AP area will be taken from all the frames and recombined into a mosaic for the final outcome image, this is a huge advantage in solar with terrible seeing conditions) on the final couple-hundred good frames you give it (and yes, benefiting from more frames to get the best AP from each frame). Local AP does a much better job of producing a superior final image of course, but it's very slow to do that on thousands of frames. So PIPP is a great way to cull the bulk of bad frames, then only feed the top contrast frames into AS!3. Saves so much time. Results are the same. Just a more efficient way to use each software together hammer-and-anvil style. Since we don't need a lot of frames in solar to stack, 40~60 is plenty, some go for hundreds or thousands (for their own reasons), this is a rapid way to get great data rapidly. And of course, is of highest interest for high resolution imaging (this is overkill for course image scale full discs that are undersampled).marktownley wrote: ↑Fri Jun 04, 2021 5:37 am Thanks Marty, that link is very informative, for reference the (basic) laptop i'm using is a i7 2.4Ghz 8Gb 256Gb. PC specs aside, my key takeaway here is that PIPP sorts frames using a global quality estimation (based on a single 'whole' frame'), AS3 does it's quality estimation based on 'local' alignment points.
This makes sense when I reflect on the resultant quality graph in AS3 as it can be quite 'up and down'. I'm saving 500 frames from PIPP but only stacking 96/128/256 in AS3 (depending on the quality of data), so, maybe there is some mileage in giving AS3 more frames to play with than used in the final stack size?
I'm capturing 4000 frames Rainer when i'm fighting the limit of seeing with the maximum aperture - within that 4000 there will be a hundred or so good ones that I can stack. If i'm using an aperture that is less than that dictated by seeing (eg Ha full disk) then I don't capture anywhere near that many. 4000 frame captures are only for the realms of hi-res work.
Very best,
- Rusted
- Way More Fun to Share It!!
- Posts: 1738
- Joined: Fri Jan 19, 2018 11:00 am
- Location: Central Denmark
- Has thanked: 8016 times
- Been thanked: 1938 times
- Contact:
1
http://fullerscopes.blogspot.dk/
H-alpha: Baader 160mm D-ERF, iStar 150/10 H-alpha objective, 2" Baader 35nm H-a, 2" Beloptik KG3,
Lunt 60MT etalon, Lunt B1200S2 BF, Assorted T-S GPCs or 2x "Shorty" Barlow, ZWO ASI174.
H-alpha: Baader 160mm D-ERF, iStar 150/10 H-alpha objective, 2" Baader 35nm H-a, 2" Beloptik KG3,
Lunt 60MT etalon, Lunt B1200S2 BF, Assorted T-S GPCs or 2x "Shorty" Barlow, ZWO ASI174.