Half duration video recordings in FireCapture and SER files

this is the main message area for anything solar :)
Post Reply
User avatar
H-Alpha
Almost There...
Almost There...
Posts: 598
Joined: Sun Feb 21, 2021 2:19 am
Location: Vouliagmeni (South Athens suburbs), Greece
Has thanked: 1674 times
Been thanked: 982 times

Half duration video recordings in FireCapture and SER files

Post by H-Alpha »

Dear all,

Yesterday I received my Astrosolar and ASI290!!! (you know the feelings with our new toys everyone... :-) ), so after some test yesterday night, today I made my first captures with Firecapture 2.6 and had two problems and two question:

1) I made various captures (AVI) of 30 second at the highest FPS rate of about 170. However, although during the capture there were no problems and no errors appeared, the videos had duration of 13 or 14 seconds! Any idea how to resolve this?

2) Trying to reduce the frames to 400 per capture (those of the highest quality), PIPP made selects 400 frames but for some reason adds 300 output duplicate frames, it says. What I am doing wrong?

3) Looking at the previous post viewtopic.php?f=4&t=30931 I realize that many if not all of you make video captures at SER format. Why is it better?

4) Is the PIPP SER player the good one to use?

Thank you all for any help you may provide,
Alexandros


User avatar
Montana
Librarian
Librarian
Posts: 34559
Joined: Mon Oct 17, 2011 5:25 pm
Location: Cheshire, UK
Has thanked: 17666 times
Been thanked: 8787 times

Re: Half duration video recordings in FireCapture and SER files

Post by Montana »

Hi Alessandros,
Are you sure you had a limit of 30 seconds set, or a limit of the number of frames in Firecapture? sounds like you had a limit of frames set and so if the frame rate is fast it will be shorter than 30 seconds.

No idea about PIPP as never used it.

I use SER only because I capture in 16bit. If you capture in AVI it is limited to 8 bit. No other reason and 8 bit is perfectly fine.

Alexandra


User avatar
marktownley
Librarian
Librarian
Posts: 42269
Joined: Tue Oct 18, 2011 5:27 pm
Location: Brierley Hills, UK
Has thanked: 20424 times
Been thanked: 10243 times
Contact:

Re: Half duration video recordings in FireCapture and SER files

Post by marktownley »

Hi there.

Like Alexandra says, sounds like there is a limit set somewhere. No idea about PIPP, don't use it. I only ever capture in 8 bit avi, I see no benefit at all in capturing in 16 bit.

Mark


Image
http://brierleyhillsolar.blogspot.co.uk/
Solar images, a collection of all the most up to date live solar data on the web, imaging & processing tutorials - please take a look!
User avatar
H-Alpha
Almost There...
Almost There...
Posts: 598
Joined: Sun Feb 21, 2021 2:19 am
Location: Vouliagmeni (South Athens suburbs), Greece
Has thanked: 1674 times
Been thanked: 982 times

Re: Half duration video recordings in FireCapture and SER files

Post by H-Alpha »

Hi Alexandra and Mark!

Thanks for your input.
I checked the logs of Firecapture and I do record for 30 seconds. I had put "No limit". nevertheless, I observed that the FPS rate drops periodically and for some seconds no frames at all are recorded/captured! this is why at the end I have much less seconds and frames... If you have any other idea... Could it be that the laptop is not so fast?

Mark, what are you using instead of PIPP to prepare your videos for AS! ?
Alexandra, which SER player are you using?

Thanks again,
Alexandros


Alexandros
Lunt 130MT+1800BF, C8 Ultima PEC+AstroSolar, Skywatcher Mount EQ6-R Pro
Baader Solar Prism, ZEISS Abbe Barlow 2x, Celestron Barlow 2x Ultima Series
ZWO ASI290MM, ZWO ASI1600MM Pro,
Greece
User avatar
marktownley
Librarian
Librarian
Posts: 42269
Joined: Tue Oct 18, 2011 5:27 pm
Location: Brierley Hills, UK
Has thanked: 20424 times
Been thanked: 10243 times
Contact:

Re: Half duration video recordings in FireCapture and SER files

Post by marktownley »

H-Alpha wrote: Wed Mar 03, 2021 8:47 pm Mark, what are you using instead of PIPP to prepare your videos for AS! ?
Nothing. I just run them straight through As3.


Image
http://brierleyhillsolar.blogspot.co.uk/
Solar images, a collection of all the most up to date live solar data on the web, imaging & processing tutorials - please take a look!
User avatar
MalVeauX
Way More Fun to Share It!!
Way More Fun to Share It!!
Posts: 1858
Joined: Tue May 09, 2017 7:58 pm
Location: Florida
Has thanked: 1171 times
Been thanked: 1360 times

Re: Half duration video recordings in FireCapture and SER files

Post by MalVeauX »

Heya,

It would help to know more about the PC you're using, components, media that you're writing this data to, etc.

1) When you capture, there are log files for each capture. Share them? This will help immensely to know exactly what you had set. If you were dropping frames, it will result in a shorter video (frame count is low) despite capturing for 30 seconds or other set values. It will be in the log file. This will point to everything to help diagnose. You can attach the log files here as attachments (below post window, options/attachments tabs).

2) In PIPP, if you turn off all features (no debeyer, no stabilization, etc), that's good. But when it comes to the Quality Options, checking the enable quality estimation box enables it, reordering the frames is your preference (not needed), and the 3rd option is the number of frames to keep (set the number you want). To the right side of this window, there's an "Estimate Quality Weighting." This is adding frames and doubling up on some frames. Uncheck this and that will solve the issue presented in this bullet.

3) The container file you choose to capture to doesn't matter. You can use SER, AVI, MP4, whatever you want assuming you have the codec support. The purpose of SER is because it has unlimited file size (unlike AVI) and because it stores date and time for the capture per frame, and this data is embedded in the container so that other software (like WinJupos or Impact Detect) can use it to precisely time (and confirm) data happening. This is most commonly important again for derotation but more specifically for impact detection projects for Saturn & Jupiter for confirming such instances. There's no inherent advantage to SER for solar imaging, so its fine to capture AVI. Just make sure you're capturing uncompressed and don't output a compressed version from PIPP if you do that.

4) PIPP's ser player is pretty common. There are not any serious players for SER as its primarily meant to simply be data container with time stamp per frame for the purpose of planetary imaging and impact detection and derotation. You can use AVI and use any AVI playback software to review your files if you wish.

You don't have to use PIPP, but if you generate large piles of data, it is faster to pre-process with PIPP first to take your potentially 5k to 10k frame piles down to a few hundred frames of best quality with PIPP first. It is indeed faster than using AS!3 for the entire pre-processing. It will shave minutes of time off pre-processing doing PIPP to cull the massive frame count first and then put the best frames through AS!3 to then further quality estimate and limit the rest. But you don't have to. If you're having issues, you can simplify by using just AS!3 for all of it, it will just add a few minutes of time.

Very best,


User avatar
H-Alpha
Almost There...
Almost There...
Posts: 598
Joined: Sun Feb 21, 2021 2:19 am
Location: Vouliagmeni (South Athens suburbs), Greece
Has thanked: 1674 times
Been thanked: 982 times

Re: Half duration video recordings in FireCapture and SER files

Post by H-Alpha »

Many thanks Marty for the rich and detailed answers!

1) You are absolutely right. I read a lot in the meanwhile in cloudy nights about the reasons and mainly by increasing the amount of RAM available for Firecapture (in Heapsize of Generall FC settings) my average FPS increased very significantly. From some 60 FPS this morning, now I get some 140 FPS as you can see in the attached Logs, which is excellent but still not the maximum 170 FPS of the camera. My question is: Is anyone using ASI cameras (or even better ASI290 like you Marty) getting AVERAGE FPS of ~170 FPS? Is this possible so that I keep trying to maximize it?

2) That was it! Thanks a lot. I unchecked "Estimate Quality Weighting" and no duplicates anymore! All the rest was already unchecked (apart quality) according to your previous advice.

3) and 4) Many thanks regarding PIPP and SER

Yes, I am using PIPP for the reasons you mention following your previous advice, to create much smaller files of the best 400 frames instead of 4-5k frames originally captured, to minimize AS process time.

Best wishes,
Alexandros
Attachments
2021-03-04-0127_5-RGB-Sun WL.txt
(809 Bytes) Downloaded 22 times
2021-03-03-1409_0-RGB-Sun WL.txt
(809 Bytes) Downloaded 18 times


User avatar
MalVeauX
Way More Fun to Share It!!
Way More Fun to Share It!!
Posts: 1858
Joined: Tue May 09, 2017 7:58 pm
Location: Florida
Has thanked: 1171 times
Been thanked: 1360 times

Re: Half duration video recordings in FireCapture and SER files

Post by MalVeauX »

H-Alpha wrote: Thu Mar 04, 2021 2:02 am Many thanks Marty for the rich and detailed answers!

1) You are absolutely right. I read a lot in the meanwhile in cloudy nights about the reasons and mainly by increasing the amount of RAM available for Firecapture (in Heapsize of Generall FC settings) my average FPS increased very significantly. From some 60 FPS this morning, now I get some 140 FPS as you can see in the attached Logs, which is excellent but still not the maximum 170 FPS of the camera. My question is: Is anyone using ASI cameras (or even better ASI290 like you Marty) getting AVERAGE FPS of ~170 FPS? Is this possible so that I keep trying to maximize it?

2) That was it! Thanks a lot. I unchecked "Estimate Quality Weighting" and no duplicates anymore! All the rest was already unchecked (apart quality) according to your previous advice.

3) and 4) Many thanks regarding PIPP and SER

Yes, I am using PIPP for the reasons you mention following your previous advice, to create much smaller files of the best 400 frames instead of 4-5k frames originally captured, to minimize AS process time.

Best wishes,
Alexandros
Heya,

The heap size needs to be anywhere from 2Gb to 4Gb, it will easily be enough to handle the throughput of the IMX290 sensor. The order of data is that it will capture the sensor output onto the IMX290's own high speed RAM (cache memory) and then it outputs it over the USB3 to system bus and is accepted into your system RAM (heap size is going to be this) which is also fast, this way you don't lose frames and it can build up in your RAM, while it writes at much slower speeds to your hard drive or SSD (prefer SSD for faster small file write speed).

170 FPS is easily obtainable with humble hardware in FireCapture.

Example in FireCapture 2.6 & 2.7 Beta:

Code: Select all

ASI290MM (IMX290LQR)

IMX290LQR Sensor Specs from ZWO

1936x1096 170 fps
320x240 737.5 fps
Here's my platform:

Code: Select all

FireCapture v2.6  Settings
------------------------------------
Camera=ZWO ASI290MM
Duration=60.000s
Frames captured=10202
File type=SER
ROI=1936x1096
FPS (avg.)=170
Shutter=1.000ms
Gain=473 (78%)

FireCapture v2.6  Settings
------------------------------------
Camera=ZWO ASI290MM
Duration=60.000s
Frames captured=43665
File type=SER
ROI=320x240
FPS (avg.)=727
Shutter=1.000ms
Gain=473 (78%)
Here's with & without a flat overlay (real time flat overlay) in FireCapture (thanks to its new multi-core support functions in 2.7 Beta:

Code: Select all

NO FLAT
FireCapture v2.7 02 BETA Settings
------------------------------------
Camera=ZWO ASI290MM
Filter=L
Duration=60.011s
Frames captured=10203
File type=SER
Bit depth=8bit
ROI=1936x1096
FPS (avg.)=170
Shutter=1.000ms
Gain=178 (29%)
HighSpeed=on
USBTraffic=100
eADU=0.248
Limit=60 Seconds

Code: Select all

WITH FULL FLAT OVERLAY REAL TIME
FireCapture v2.7 02 BETA Settings
------------------------------------
Camera=ZWO ASI290MM
Duration=59.987s
Frames captured=10198
File type=SER
Bit depth=8bit
ROI=1936x1096
FPS (avg.)=170
Shutter=1.000ms
Gain=231 (38%)
HighSpeed=on
USBTraffic=100
eADU=0.248
Limit=60 Seconds
Flat=ZWOASI290MM_1936x1096__8bit_Exp=1.00ms_T=31°C_Frames=21_2020-16-11_110844.flat
You can see full 170 FPS in every instance even with flat overlay for an entire minute with no dropped frames.

The hardware is basic, it's just an AM4 motherboard (B450 chipset), Athlon 3000G CPU (cheap dual core), DDR4 3000mhz RAM (16GB) and a 1TB SSD (Samsung EVO). Very cheap. Less than $300 built this desktop.

What hardware are you using?

Very best,


User avatar
rsfoto
Way More Fun to Share It!!
Way More Fun to Share It!!
Posts: 6159
Joined: Mon Jun 18, 2012 8:30 pm
Location: San Luis Potosi, México
Has thanked: 9401 times
Been thanked: 5561 times

Re: Half duration video recordings in FireCapture and SER files

Post by rsfoto »

Hi Alexandros,

You already have got a lot of answers.

Just make a simple calculation of the streaming speed you need when streaming 170 frames per second at 16 bit

1936 x 1096 x 16 x 170 = 5,771,448,320 bits per second or 721,431,040 bytes per second or ~ 721 MB per second you want to write onto your Hard disk or SSD ... and if you stream in 8 bit, it would be the half of the above values.


regards Rainer

Observatorio Real de 14
San Luis Potosi Mexico

North 22° West 101°
User avatar
H-Alpha
Almost There...
Almost There...
Posts: 598
Joined: Sun Feb 21, 2021 2:19 am
Location: Vouliagmeni (South Athens suburbs), Greece
Has thanked: 1674 times
Been thanked: 982 times

Re: Half duration video recordings in FireCapture and SER files

Post by H-Alpha »

Thanks a lot Marty and Rainer,

It is obvious from your logs that my system can do better, but for the moment cannot locate the problem. I am stacked at about 140FPS on average. Nevertheless, I have observed what is happening, but first please let me list my laptop's components:
LENOVO, Win10
Intel(R) Core i5.3320M [email protected] Gh
RAM 8 Gb
SSD Kingston SUV5004806 (480 Giga, 350 free)
The SSD has an up to 520/500MB/s Sequential Read/Write (according to Kingston)
Therefore, According Rainer at 8 bit (I have not checked the 16 bits, but also in my video player I sea it is 8 bits), I should have no problem at all at the SSD side!

When I start to capture all goes well with a FPS rate of about 180. At about 5 seconds o capture the ventilator starts being heard louder (and steadily increases its effort), at 10 seconds the RAM (in FC bar) starts decreasing and at about 15-20 seconds the FPS drop to 120 or 80. Then the RAM increases again until almost the 30 seconds when it starts dropping again. Recordings up to 5 seconds have average 160-5 FPS, 10 seconds have 155, 20 seconds have 150 and 30 seconds 140... Apparently, some buffer is not well "tuned" and gets overloaded... This is my guess. Any ideas?

All the best,
Alexandros


User avatar
rsfoto
Way More Fun to Share It!!
Way More Fun to Share It!!
Posts: 6159
Joined: Mon Jun 18, 2012 8:30 pm
Location: San Luis Potosi, México
Has thanked: 9401 times
Been thanked: 5561 times

Re: Half duration video recordings in FireCapture and SER files

Post by rsfoto »

H-Alpha wrote: Thu Mar 04, 2021 5:07 pm Thanks a lot Marty and Rainer,

It is obvious from your logs that my system can do better, but for the moment cannot locate the problem. I am stacked at about 140FPS on average. Nevertheless, I have observed what is happening, but first please let me list my laptop's components:
LENOVO, Win10
Intel(R) Core i5.3320M [email protected] Gh
RAM 8 Gb
SSD Kingston SUV5004806 (480 Giga, 350 free)
The SSD has an up to 520/500MB/s Sequential Read/Write (according to Kingston)
Therefore, According Rainer at 8 bit (I have not checked the 16 bits, but also in my video player I sea it is 8 bits), I should have no problem at all at the SSD side!

When I start to capture all goes well with a FPS rate of about 180. At about 5 seconds o capture the ventilator starts being heard louder (and steadily increases its effort), at 10 seconds the RAM (in FC bar) starts decreasing and at about 15-20 seconds the FPS drop to 120 or 80. Then the RAM increases again until almost the 30 seconds when it starts dropping again. Recordings up to 5 seconds have average 160-5 FPS, 10 seconds have 155, 20 seconds have 150 and 30 seconds 140... Apparently, some buffer is not well "tuned" and gets overloaded... This is my guess. Any ideas?

All the best,
Alexandros
Hi,

Everything sounds very nice but the bottleneck can be the internal USB design ... and here you can not do anything ... It is a question of how they designed their boards.

Theoretically USB 3.0 is 5 Gbit but that is pure theory.

On the other side why do you need 30 seconds at 170 fps which gives you about 5100 frames ? When processing the video you will get nuts :lol:

Acquire max 1000 frames and process them or make a series of 250, 500 and 1000 frames. Process them all equally and then compare results.

BTW my poor experience tell me that setting a speed ox max 30 fps per second and record about 1000 frames will equalize better the waves of atmospheric turbulence. You will never catch the sharp wave. Sit in front of your monitor and what how the turbulence passes by and you will learn a lot about his phenomena

All just IMHO


regards Rainer

Observatorio Real de 14
San Luis Potosi Mexico

North 22° West 101°
User avatar
H-Alpha
Almost There...
Almost There...
Posts: 598
Joined: Sun Feb 21, 2021 2:19 am
Location: Vouliagmeni (South Athens suburbs), Greece
Has thanked: 1674 times
Been thanked: 982 times

Re: Half duration video recordings in FireCapture and SER files

Post by H-Alpha »

Gracias Rainer!
Will have to live with the slightly lower rate and will try to make the tests you propose.
Having only the C8 so far, it won't be easy anyway (until I get the LUNT80 or 100). Nevertheless, the first test of today gives some hope.
I think I will upload my fist WL photo in another post for comments.
best wishes,
Alexandros


Alexandros
Lunt 130MT+1800BF, C8 Ultima PEC+AstroSolar, Skywatcher Mount EQ6-R Pro
Baader Solar Prism, ZEISS Abbe Barlow 2x, Celestron Barlow 2x Ultima Series
ZWO ASI290MM, ZWO ASI1600MM Pro,
Greece
User avatar
MalVeauX
Way More Fun to Share It!!
Way More Fun to Share It!!
Posts: 1858
Joined: Tue May 09, 2017 7:58 pm
Location: Florida
Has thanked: 1171 times
Been thanked: 1360 times

Re: Half duration video recordings in FireCapture and SER files

Post by MalVeauX »

Heya,

Sounds like laptop hardware to me. The system bus is going to be the limit regardless, and most laptop hardware will not have the same throughput as desktop hardware. Your hardware should be able to handle this throughput. You could try testing the write speed of your SSD. You can test the SSD health with Crystal Disc Info (free) and you can bench it with various times/file sizes with Crystal Disc Mark (also free); set it to 10Gb to replicate the kind of size file you're using. It also may be the SSD because of its lack of cache memory and perhaps it has poor small file sequential write speed. But somewhere from your system bus receiving the data from the camera, to your system RAM (with large heap size) to the bus again and then to the SSD is hitting not just a bottle neck but a throttling issue. If you are hearing your fan kick on immediately, then something is getting very hot. These systems will throttle down everything to keep the heat down automatically. Since the CPU is not doing any work during this, it may be the memory or system bus itself or even the SSD, or all of them, getting too hot from the throughput.

I realize a laptop is more convenient for almost everyone. However, if you want to max out your cameras in the future, I recommend a mini ITX form factor desktop build. They're inexpensvie and will not have these restrictions.

Very best,


User avatar
H-Alpha
Almost There...
Almost There...
Posts: 598
Joined: Sun Feb 21, 2021 2:19 am
Location: Vouliagmeni (South Athens suburbs), Greece
Has thanked: 1674 times
Been thanked: 982 times

Re: Half duration video recordings in FireCapture and SER files

Post by H-Alpha »

Thanks Marty,
Will test the SSD as you propose.
Just before reading your new post, I found something that made me reach 165FPS :-)

I checked 'Use RAM Buffer' in the General Settings of FC and add the maximum value that it could accept. I tested it with the Test button and saw that it was ok. Then while capturing I realized that the fan was not anymore starting or not running as much as before, and after checking the results I saw that I was always above 160FPS! :-)

Thanks once more,
Alexandros


Alexandros
Lunt 130MT+1800BF, C8 Ultima PEC+AstroSolar, Skywatcher Mount EQ6-R Pro
Baader Solar Prism, ZEISS Abbe Barlow 2x, Celestron Barlow 2x Ultima Series
ZWO ASI290MM, ZWO ASI1600MM Pro,
Greece
User avatar
MalVeauX
Way More Fun to Share It!!
Way More Fun to Share It!!
Posts: 1858
Joined: Tue May 09, 2017 7:58 pm
Location: Florida
Has thanked: 1171 times
Been thanked: 1360 times

Re: Half duration video recordings in FireCapture and SER files

Post by MalVeauX »

Sounds good, 160 FPS is plenty good result.

Very best,


Post Reply