CS page

Jan 12 2014, at Pachon. 

Have powered up mac mini within the Illinois aeronomy building. Set networking for hardwired ethernet to

IP 139.229.19.220

submet mask 255.255.255.224

router 139.229.19.193

we can also use IP address 139.229.19.221 if need be. 

local access is ssh christopherstubbs@139.229.19.220, cat4apple

Chris,

Please set up your computer using the following parameters:

Router  139.229.19.193

Mask: 255.255.255.224

DNS: 139.229.15.130, 139.229.2.3

IP 139.229.19.220

If you need additional IP number, 139.229.19.221 can also be used.

In terms of remote access to ALO computers, you also need to obtain information about firewall access from Ron Lambert  rlambert@ctio.noao.edu, who is in the process of reconfiguring the firewall at ALO.

Let me know if you have any questions,

Alan

-- 
Alan Liu, Ph.D.
Associate Professor
Physical Sciences, Daytona Beach
Embry-Riddle Aeronautical University

Daytona Beach, Florida
Office: +1-386-226-6538
email; alan.liu@erau.edu

here are the scripts that run, as of Jan 12:

startnight.sh (takes NN frames as command line argument)

server:~ christopherstubbs$ cat startnight.sh

#!/bin/bash

# modified Dec 12 2013 to allow N frames set from command line, overwrites what was set in initialize.sh

export maxframes=$1

echo first arguent is $1

source ~/initialize.sh

echo maxframes is $1

while [ $framecounter -lt $maxframes ]; do

echo sleeping for $pauseinterval

sleep $pauseinterval

gphoto2 --set-config=/main/capturesettings/shutterspeed=1

# this next trick pads out the framecounter to 4 digits for decent file names

export paddedcounter=`printf  "%04i\n" $framecounter`

gphoto2 --capture-image-and-download --filename $dirpath/$dirname/$dirname.$paddedcounter.short.cr2

let framecounter=framecounter+1

export paddedcounter=`printf  "%04i\n" $framecounter`

gphoto2 --set-config=/main/capturesettings/shutterspeed=10

export paddedcounter=`printf  "%04i\n" $framecounter`

gphoto2 --capture-image-and-download --filename $dirpath/$dirname/$dirname.$paddedcounter.long.cr2

let framecounter=framecounter+1

done

# at this stage, we're done collecting images for the night. Wrap things up...

ls $dirpath/$dirname/*.cr2 | wc | awk '{print $1," images collected"}' >> $dirpath/$dirname/$dirname.log

# now do image conversion to fits, and run some initial analysis.

echo making fits files at `date`

source ~/makefits.sh

echo extracting sky values at `date`

source ~/getsky.sh

echo doing photometry at `date`

source ~/photometry.sh

# copy result files to Amazon Web Services machine

echo copying files to aws at `date`

scp -i ~/aws/aws1.pem.txt -r $dirpath/$dirname/$dirname.* ec2-user@54.200.60.175:~/data/

echo compressing images at `date`

cd $dirpath/$dirname

gzip B/*.fits

gzip G/*.fits

gzip R/*.fits

gzip M/*.fits

gzip CR2/*.cr2

echo moving the log file to appropriate directory at `date`

mv ~/cronlog $dirpath/$dirname/$dirname.cronlog

initialize.sh

server:~ christopherstubbs$ cat initialize.sh

#!/bin/bash

# try to fix up user issues for crontab

source /Users/christopherstubbs/.bash_profile

# set up directory names and data paths

export datadisk=/dev/disk1s2

export dirpath=/Users/christopherstubbs/data

export dirname=`date +"ut%m%d%y"`

export thismonth=`date +"%m"`

# create tonight's directory

mkdir $dirpath/$dirname

cd $dirpath/$dirname

date | awk '{print "started at "$0 }'>> $dirname.log

df -H | grep $datadisk | sed s/G//g | awk '{print $4," GB left on data disk"}' >> $dirname.log

export spaceleft=`df -H | grep $datadisk | sed s/G//g | awk '{print $4}'`

# fix this later

#if ($spaceleft>20); then

# mail -s "allsky camera lots space!" stubbs@physics.harvard.edu

#fi

# wipe out any existing connections to camera

killall PTPCamera

# set up environment variables for camera

export framecounter=0001

export xctr=2875

export yctr=1920

# prefix for images, gets framecounter.cr2 appended

export imageprefix=$dirname.

# interval after end of last image before starting next one, in seconds

export pauseinterval=50 

# configure aspects of camera that won't change

# synch camera datetime to ntp-served acquisition computer value, in UT

gphoto2 --set-config syncdatetime=1

gphoto2 --set-config iso=1600

gphoto2 --set-config aperture=2.8

cd ~

photometry.sh: 

server:~ christopherstubbs$ cat photometry.sh

#!/bin/bash

# performs quick photometry on each image, using tphot

cd $dirpath/$dirname/M

rm *.phot

rm *.nstars

for i in *.M.fits; do tphot $i -out `basename $i .fits`.phot; done

# how many stars in each image? 

for i in *.M.phot; do wc $i  | awk '{print ($1-1)}'  >> $dirname.nstars ; done

mv $dirname.nstars ..

cd $dirpath/$dirname/B

rm *.phot

for i in *.B.fits; do tphot $i -out `basename $i .fits`.phot; done

cd $dirpath/$dirname/G

rm *.phot

for i in *.G.fits; do tphot $i -out `basename $i .fits`.phot; done

cd $dirpath/$dirname/R

rm *.phot

for i in *.R.fits; do tphot $i -out `basename $i .fits`.phot; done

cd ..

rm temp

# add extra line to nstars file to accommodate header line coming up

echo " " > temp2

cat $dirname.nstars >> temp2

mv temp2 $dirname.nstars

rm temp2

# add another column to the obslog file

paste $dirname.obslog $dirname.nstars >> temp

mv temp $dirname.obslog

# separate out long and short exposures

echo "#image     BIAS EXPTIME MJD-OBS                       M        B       G       R nstars" > $dirname.short.obslog

grep short $dirname.obslog >> $dirname.short.obslog

echo "#image     BIAS EXPTIME MJD-OBS                       M        B       G       R nstars" > $dirname.long.obslog

grep long $dirname.obslog >> $dirname.long.obslog

Here is sensitivity curve for Canon 5D color bands:

FOV of 15mm Sigma fisheye lens is as follows:

in long direction: 147.5 degrees, or 73 degrees off zenith

in short direction: 94.3 degrees, or 47 degrees off zenith

corner-to-corner: 184 degrees, or horizon to horizon.

We're currently oriented with EW in long direction, and NS in short direction. So we don't quite get to celestial pole in the S but we do get to the equator in the N

Last night we were successful in getting the system focused, and took a directory full of data. But it was started after midnight and so the directory name and files names are both wrong. It should be listed as ut011214 but instead all are listed as ut011314. So need to fix this. Will likely just rename directory. 

April 12 2014

Optimizing SE parameters for monochrome Canon images, with MC.

we have 4 test images, will try to find good settings for all 4. 

Working first on image ut030114.0170. First crack has PSF FWHM ranging from ~1 at center to ~2.2 at edge of frame. Chose Gaussian 2 pix FWHM kernel for convolution. 

with this file, we got 3165 lines in test.cat:

# Default configuration file for SExtractor 2.3b2

# EB 2003-02-07

#

#-------------------------------- Catalog ------------------------------------

CATALOG_NAME    test.cat        # name of the output catalog

CATALOG_TYPE    ASCII_HEAD      # "NONE","ASCII_HEAD","ASCII","FITS_1.0"

                                # or "FITS_LDAC"

PARAMETERS_NAME allsky.param   # name of the file containing catalog contents

#------------------------------- Extraction ----------------------------------

DETECT_TYPE     CCD             # "CCD" or "PHOTO"

FLAG_IMAGE     FLAG.FITS        # filename for an input FLAG-image

DETECT_MINAREA  2               # minimum number of pixels above threshold

DETECT_THRESH   2.5             # <sigmas> or <threshold>,<ZP> in mag.arcsec-2

ANALYSIS_THRESH 2.5             # <sigmas> or <threshold>,<ZP> in mag.arcsec-2

FILTER          Y               # apply filter for detection ("Y" or "N")?

FILTER_NAME     allsky.conv    # name of the file containing the filter

DEBLEND_NTHRESH 2              # Number of deblending sub-thresholds

DEBLEND_MINCONT 0.05           # Minimum contrast parameter for deblending

CLEAN           Y               # Clean spurious detections? (Y or N)?

CLEAN_PARAM     1.0             # Cleaning efficiency

MASK_TYPE       CORRECT         # type of detection MASKing: can be one of

                                # "NONE", "BLANK" or "CORRECT"

#------------------------------ Photometry -----------------------------------

PHOT_APERTURES  2,4,6,8               # MAG_APER aperture diameter(s) in pixels

PHOT_AUTOPARAMS 2.5, 3.5        # MAG_AUTO parameters: <Kron_fact>,<min_radius>

SATUR_LEVEL     15000.0         # level (in ADUs) at which arises saturation

MAG_ZEROPOINT   10.0             # magnitude zero-point

MAG_GAMMA       4.0             # gamma of emulsion (for photographic scans)

GAIN            1.0             # detector gain in e-/ADU

PIXEL_SCALE     1.0             # size of pixel in arcsec (0=use FITS WCS info)

#------------------------- Star/Galaxy Separation ----------------------------

SEEING_FWHM     2             # stellar FWHM in arcsec

STARNNW_NAME    default.nnw     # Neural-Network_Weight table filename

#------------------------------ Background -----------------------------------

BACK_SIZE       60              # Background mesh: <size> or <width>,<height>

BACK_FILTERSIZE 10               # Background filter: <size> or <width>,<height>

BACKPHOTO_TYPE  LOCAL          # can be "GLOBAL" or "LOCAL"

#------------------------------ Check Image ----------------------------------

CHECKIMAGE_TYPE -OBJECTS            # can be one of "NONE", "BACKGROUND",

                                # "MINIBACKGROUND", "-BACKGROUND", "OBJECTS",

                                # "-OBJECTS", "SEGMENTATION", "APERTURES",

                                # or "FILTERED"

CHECKIMAGE_NAME check.fits      # Filename for the check-image

#--------------------- Memory (change with caution!) -------------------------

MEMORY_OBJSTACK 2000            # number of objects in stack

MEMORY_PIXSTACK 200000          # number of pixels in stack

MEMORY_BUFSIZE  2048            # number of lines in buffer

#----------------------------- Miscellaneous ---------------------------------

VERBOSE_TYPE    NORMAL          # can be "QUIET", "NORMAL" or "FULL"

But we're missing many faint stars. Try reducing required pixels above threshold. Dropping from 2 to 1. Also made images with bias level of 2048 subtracted. 

Really weird- SE is missing most faint objects. Reducing threshold to 0.1 sigma didn't help, nor did playing with sky estimation parameters. Looking at output file, seems we are way overestimating the noise in photometry, so just not getting over threshold. So... let's try broadening the PSF with a convolution before sending image to SE. 

Ahah! the issue seems to be that image files need to be converted to reals before passing to SE. Then we get fairly rational behavior. 

Since the big deal seems to be converting images to reals, made two Chile test images that are reals, by multiplying by 1.001. Load those frames into our test suite.

Full SE photometry files on 170 and 650

ut030114.0650.cat

ut030114.0170.cat

56717.5972338.cat

56717.2500116.cat

allsky.sex

allsky.param

allsky.conv

to convert SE output files into stripped-down x,y,phi,dphi files for JT, do

grep -v "#" filename.cat | awk '{print $2, $3, $6, $7}' > filename.dat

this grabs x,y,flux_auto, flux_auto error. Resulting data files with no header information:

June 1 2014

We need to omit variable stars. Use this list as a starting point:

variable_stars.dat

Also, here is a catalog with magnitudes in SDSS system:

Tycho_SDSS_bright.barsep.dat