Raw images - a technical question

this is the main message area for anything solar :)
Post Reply
Bruce Girrell

Raw images - a technical question

Post by Bruce Girrell »

The basic question here is "Do any of the commercially available astro imaging programs provide a true raw image format?"

The following should not matter as far as true raw images go, but I'm sure someone will ask, so here are some specifics
Two different cameras are being used, a QHY 163M and a ZWO ASI 174MM*
Both are used at 16-bit capture (both use 12-bit A/D converters)
I am using SharpCap to capture the images. I can't get FireCapture to work. I have not tried Sequence Generator Pro or Maxim DL and I have not looked for other astro imaging programs.

By "true raw" I am talking about a record of the output of the imaging chip after A/D conversion but before anything else. For digital SLRs, for example, if I capture a RAW image, it represents the luminance (and color) values as the imaging chip detected them. It doesn't, for example, matter what white balance I use because that is simply an adjustment to the color balance of the image. It does not affect how the image sensor saw the incoming photons. Therefore, I can take a raw image and apply whatever white balance I want later on in post-processing. Similarly, other computations are applied to JPEG images that don't exist in the raw image, but given the raw image, I can duplicate any of those modifications in post-processing. Since telescopes operate at a fixed aperture, please allow me to assume that the aperture of my digital SLR remains constant. Given that restriction, shutter speed and the ISO setting of my SLR determine 1) how much light falls on the image sensor and 2) how much amplification is applied to the image sensor output prior to the A/D converter. No other camera settings affect the raw image.

I have been unable to find suitable block diagrams for any astro cameras, so I'm not sure how the image is processed before it is written out. In particular, it is not clear at what point the gamma adjustment is incorporated. It would appear to me that gamma is introduced _after_ A/D conversion. As the gamma control is adjusted to its extremes, you can clearly see separation of digital values showing up as distinct bars in the image histogram. It would seem that if gamma is being applied after the ADC, then then same function could be applied in post-processing software. I also note that the FITS format, which at least one camera manufacturer considers to be raw, is affected by the gamma setting and hence, the FITS format is not true raw, as it contains digital operations on the image sensor data. Other controls, such as Offset or Brightness, appear to be added in analog and therefore do belong in the raw image.

The issue that I have with gamma, as it appears to be applied in astro imaging, is that it expands one portion of the histogram (at the expense of the remainder, of course). But solar images need help on both ends of the histogram at the same time. If I optimize the appearance of the solar disk, I lose the prominences. If I want to keep the prominences, I lose most of the detail in the solar disk.

In film photography, the gamma curve has a sigmoid appearance. Using a film gamma curve, I can expand contrast in the highlights (read "solar disk") and I can also expand contrast in the shadow areas (read "prominences"), sacrificing some of the contrast in the mid-tones. Photoshop and other image editing programs allow this double-ended gamma, so it would seem that the best option would be to record raw format images from the astro camera, then post process to achieve the desired gamma response.

So, this brings me to my original question - How do I get a true raw image out of my astro camera? I suppose I could also ask, "Is there any astro image capture program that allows a two (or more) point gamma curve?" But again, if the gamma is applied after the ADC, then no matter what it does, the same effect can be achieved in post processing of a raw image.

Any suggestions - including straightening out misperceptions or warped ideas of mine - are welcome.

Bruce G


* It should be noted that the gamma values for these two cameras are different and appear to operate in opposite directions. The QHY camera has a gamma that ranges from 0 to 2.0. You use a low gamma for prominences and a high gamma for the Sun itself. The gamma of the ZWO camera ranges from 1 to 100 and you set the value to a low number for the solar disk and to a high number for the prominences. Please be mindful of this and try to avoid wording such as "set the gamma to a lower value"


User avatar
Merlin66
Librarian
Librarian
Posts: 3970
Joined: Tue Dec 13, 2011 1:23 pm
Location: Junortoun, Australia
Has thanked: 173 times
Been thanked: 615 times
Contact:

Re: Raw images - a technical question

Post by Merlin66 »

I use the ASI under FireCapture.....
I also use AA for all my other cameras DSLR and CCD.
The Canon RAW file is manipulated by the inbuilt software before being available. This is obvious by the lack of RGGB pixels in the image, they are all converted to some colour.


"Astronomical Spectroscopy - The Final Frontier" - to boldly go where few amateurs have gone before
https://groups.io/g/astronomicalspectroscopy  
http://astronomicalspectroscopy.com
"Astronomical Spectroscopy for Amateurs" and
"Imaging Sunlight - using a digital spectroheliograph" - Springer
User avatar
PDB
Almost There...
Almost There...
Posts: 702
Joined: Mon Aug 15, 2016 4:23 pm
Location: Belgium
Has thanked: 2 times
Been thanked: 149 times

Re: Raw images - a technical question

Post by PDB »

Hello Bruce,

good question. If i remember correctly most of the cameras will do gamma adjustment in software (after adc). Think there is only one (but forgot which one, ptgrey? Others may know this) that has gamma adjustment in hardware. Can't speak for QHY, but ZWO cams have a neutral setting where no change to Gamma is done. YFor these you need to set gamma to 50 (or off in FireCapture) for this. Think they might do small change to the output by stretching the 12 (or 14 for ZWO178) bits to 16 bit, but not really sure. (Should do some testing with that again to be sure)

Regards,

Paul


User avatar
Montana
Librarian
Librarian
Posts: 34560
Joined: Mon Oct 17, 2011 5:25 pm
Location: Cheshire, UK
Has thanked: 17667 times
Been thanked: 8788 times

Re: Raw images - a technical question

Post by Montana »

It is best practice and always has been to never adjust the gamma before acquisition, acquire the full data and change the contrast to suit in post processing otherwise you will lose all manner of faint solar features. By all means reduce the gamma so that you have more contrast to focus but move it back before acquisition.

Alexandra


Bruce Girrell

Re: Raw images - a technical question

Post by Bruce Girrell »

I guess it appeared to me that I was losing some portion of my data no matter what I did. Looks like I need to spend a little more time at the computer.

Alexandra - I know what you mean. In fact it's part of what prompted my question. I was looking for a way to say to the capture software "Just give me the raw data and I'll sort it out later." My company makes instruments that perform remote sensing tasks and it has always been our philosophy to have the instruments store the raw data. You don't always know what you may want it for. One time it turned out that "noise" in the data was exquisitely detailed information. You just had to process it a bit differently to see it.

Any particular tricks for trying to bring out those proms?

Thanks all

Bruce G


User avatar
Montana
Librarian
Librarian
Posts: 34560
Joined: Mon Oct 17, 2011 5:25 pm
Location: Cheshire, UK
Has thanked: 17667 times
Been thanked: 8788 times

Re: Raw images - a technical question

Post by Montana »

Bruce I wrote a beginners guide to imaging the Sun if you want to browse it
https://solarnutcase.livejournal.com/18572.html
Alexandra


Bruce Girrell

Re: Raw images - a technical question

Post by Bruce Girrell »

Thank you Alexandra
That looks very useful


Bruce Girrell

Re: Raw images - a technical question

Post by Bruce Girrell »

Alexandra - Thanks for the recommendation on imPPG. It is much more intuitive and easier to use than, say, Astra Image, though limited to L-R deconvolution. I understand deconvolution and what PSFs are, but it is not at all clear to me when I want to use which kernel for what method in Astra Image and I found the program frustrating to use. I still have my day job, and I just don't have time to study the different deconvolution types and kernels. So, limiting to one method then isn't that big of an issue for me, though it would be nice to understand why L-R deconvolution seems to be favored by astrophotographers.

Quick question - The program recommends the number of iterations between 30 and 70, but you use 500. Is that a typo, or do you really use 500? I found a low number (50) to be better. How do you judge where to set your iteration count?

Still playing with reprocessing of older images to see what sort of improvement I can get. I use Photoshop a lot, but I had forgotten about Shadows and Highlights adjustment - just what is needed for constraining the bright stuff while bringing out the proms. Thanks for the reminder.


User avatar
GreatAttractor
Almost There...
Almost There...
Posts: 964
Joined: Sat Jun 01, 2013 1:04 pm
Location: Switzerland
Has thanked: 747 times
Been thanked: 753 times

Re: Raw images - a technical question

Post by GreatAttractor »

Hi Bruce,
I'm glad you like it.
The program recommends the number of iterations between 30 and 70
Or rather, I put that recommendation in judging by my own images. Alexandra favors higher values, so I guess it depends on the setup and the raw material it produces. Keep experimenting until you're satisfied.
but I had forgotten about Shadows and Highlights adjustment - just what is needed for constraining the bright stuff while bringing out the proms
You can probably get similar effect by manipulating the tone curve (also possible from ImPPG).


My software:
Stackistry — an open-source cross-platform image stacker
ImPPG — stack post-processing and animation alignment
My images

SW Mak-Cass 127, ATM Hα scopes (90 mm, 200 mm), Lunt LS50THa, ATM SSM, ATM Newt 300/1500 mm, PGR Chameleon 3 mono (ICX445)
Post Reply