So..... Where do you people put your server farms?

this is the main message area for anything solar :)
Post Reply
Bruce Girrell

So..... Where do you people put your server farms?

Post by Bruce Girrell »

In the basement, the garage, or do you get a divorce and take over your ex's side of the bedroom?

In just a couple hours of imaging activity yesterday, I ran through over 1/2 Terabyte of hard drive space. This can't go on, or I'll be buying a new hard drive every time that I want to take some pictures.

How do you manage your storage space? After stacking, for example, do you delete the raw files and only keep the result? Do you keep only the fraction that you stacked? What do you normally keep and what do you delete throughout your workflow?

I have already started being more careful about the capture image size, minimizing empty space in the image. But at the same time, I'm also working to increase the image size that is being sent to my camera (Barlows, e.g.) to give me more latitude in processing. With a larger image, I have less empty space in the image, leaving little room to crop, so there's not a lot to be gained there.

Compression? File formats? What are the levers that help us control the data overload?

Thanks much,
Bruce G



From yesterday:
Ha-overlay.gif
Ha-overlay.gif (11.24 MiB) Viewed 1074 times


User avatar
rsfoto
Way More Fun to Share It!!
Way More Fun to Share It!!
Posts: 6159
Joined: Mon Jun 18, 2012 8:30 pm
Location: San Luis Potosi, México
Has thanked: 9401 times
Been thanked: 5568 times

Re: So..... Where do you people put your server farms?

Post by rsfoto »

Hi Bruce,

Nice animation from White light into H-alpha :bow

May I ask what is your workflow ?

In my case for a H-alpha image session for a full disk consisting of a 9 image mosaic I shoot 9 AVI (1216x1216pixels) with each 500 images only which sums up to about 6.5GB. Every month or two I delete the original AVI.

regards Rainer


regards Rainer

Observatorio Real de 14
San Luis Potosi Mexico

North 22° West 101°
User avatar
Montana
Librarian
Librarian
Posts: 34560
Joined: Mon Oct 17, 2011 5:25 pm
Location: Cheshire, UK
Has thanked: 17667 times
Been thanked: 8789 times

Re: So..... Where do you people put your server farms?

Post by Montana »

That's a really super animation and very fine images :bow :hamster:

I used to keep every AVI/SER file and I have about 6 varying 1-4 TB hard drives in the office. However now, once a make the stacked image I delete ithe avi unless it is a really super or usual event, I am more cautious with what I keep. I know that in the future stacking software will improve and I need to keep those really good ones for re-processing, but I know I wouldn't re-process anything but the exceptional ones.

Alexandra


Bruce Girrell

Re: So..... Where do you people put your server farms?

Post by Bruce Girrell »

Thanks for the kind words

Rainer - Currently I do this:

Capture
I capture with SharpCap - I just haven't been able to get FireCapture to work right yet
The camera is a QHY 163M, with a 4656 x 3522 pixel sensor
I use 16-bit due to the huge dynamic range of image brightness
I usually take from 500 to 2000 images for each attempt. Typical seeing is poor to fair. I'm in a city, working from my back yard
I usually make multiple attempts to image each subject, varying gamma and tweaking other controls, using (or not) a Barlow, and other small changes. The gamma drives me up a wall because I can only operate on one portion of the histogram at a time, so I'm constantly trying to tweak it. I just discovered Maxim DL, which has a gamma curve like film that allows you to manipulate both ends of the histogram, but I have not tried the program yet.

Review
I review the day's captures and decide which ones I think may turn out best. Sometimes I do separate exposures for the solar disk and another for just the prominences, so I choose which images will work best together.

Processing
Sometimes PIPP to do cropping or other simple preparatory work
A small amount of contrast adjustment in Photoshop (hoping to help the following programs to find gradients easier)
Stacking with AutoStakkert!, usually using the top 5, 10, 20 and 40% of images
Deconvolution with AstraImage
Wavelet sharpening with Registax
Back to Photoshop for localized contrast enhancement, denoising (using Neat Image), maybe filtering (a 1-pixel median gets rid of a lot of that pixel-to-pixel noise), coloring if desired, and final cropping and sizing

Any suggestions are welcome. I am quite new to this field and am still very much on my learning curve.


Alexandra - I'm quickly filling up my 4 Tb drive. I guess I'll have to learn how to let go of my acquired data once I have decided to use a specific file or files from the imaging session. I'm still plagued with doubt at this point and the desire to hold on to everything until I learn the processing side of this business a bit better is strong. In reality, I should probably keep the images of the big flare and loops from Sept 10 and trash everything else. I guess if I don't get back to reprocess a file after a couple of months, I probably never will and I should delete them. Thanks for your insights.

Bruce G
Last edited by Bruce Girrell on Mon Sep 18, 2017 8:56 pm, edited 1 time in total.


User avatar
rsfoto
Way More Fun to Share It!!
Way More Fun to Share It!!
Posts: 6159
Joined: Mon Jun 18, 2012 8:30 pm
Location: San Luis Potosi, México
Has thanked: 9401 times
Been thanked: 5568 times

Re: So..... Where do you people put your server farms?

Post by rsfoto »

Hi,

Well I see this
The camera is a QHY 163M, with a 4656 x 3522 pixel sensor
¿ is it really necessary ? That is what fills up your disks. In your case I would try to bin the sensor unless you want to print huge Sun posters :-)

regards Rainer


regards Rainer

Observatorio Real de 14
San Luis Potosi Mexico

North 22° West 101°
Bruce Girrell

Re: So..... Where do you people put your server farms?

Post by Bruce Girrell »

Rainer -

> ¿ is it really necessary ?
I don't actually know. I would have to do a calculation (that I don't know how to do yet) relating the angular resolution of the telescope (Lunt LS80THa) to the size of the projected image at the image sensor. As it stands (including the Barlow), the image nearly fills the whole sensor. So for the moment I have to say yes, this is necessary. I'm not trying to make large prints, but I am trying to retain as much detail as possible.

I have always leaned toward doing as much as possible in the analog domain before switching to digital representation. Given quality optics, optical zoom is always better than digital. So to me a larger image at the image sensor gives me more processing latitude. I don't know how much empty magnification may be imposed by the Barlow, but I have been very impressed both by the sharpness of this little refractor and the capabilities of astro-imaging software to extract information out of mush. So I'm hesitant to remove the Barlow and go to a smaller sensor (especially since the camera that I have with a smaller sensor also has significantly larger pixels).


Post Reply