Reducing OSELOTS Spectral Data

On this page, we provide instructions on reducing a single night of OSELOTS data.

We do not provide a detailed documentation of the reduction code on this page - the user should look in the code itself for those details.

We do describe the results of running the analysis here.

The reduction code is available in the SpepctrographAnalysis repository on the Stubbs group GitHub page:

https://github.com/stubbslab/SpectrographAnalysis

In addition to the custom code in that GitHub repository, you also need to install the following 3rd party Python libraries (pip install XXX should do the trick):

```
numpy 
```
```
matplotlib
```
```
scipy 
```
```
pyfits 
```
```
astropy 
```
```
PyAstronomy
```

Starting the reductions

The entire core reduction process can be executed using a single Python script: ReduceOSELOTSNight.py

The script is run in Python, from the command line. The user must provide only one argument, the observing night of the data they want to analyzie, formatted as YYYY_MM_DD:

$ python ReduceOSELOTSNight.py 2022_06_29

The code will then ask the user to answer some questions about which data they wish to analyze, and then start the (somewhat lengthy) reduction process. We describe those questions below.

Running the Reductions

*** FIRST TIME USERS SHOULD UPDATE THE dir_root variable in the ReduceOSELOTSNight.py code to be where they stash OSELOTS data ***

Once the user has started the reduction, the code will ask them a series of questions about which data they want to analyze.

The questions will always provide defaults. To accept the defaults, hit [RETURN] without any other entries.

The questions are written to be self-explanatory, but we provide some extra description here. Note the code will print other information, which we do not list here:

$ python ReduceOSELOTSNight.py 2022_06_29

```
I am going to analyze data in this directory: /Users/sashabrownsberger/Documents/Harvard/physics/stubbs/skySpectrograph/data/2022_06_29/  Is that okay (y, Y, yes, Yes, YES, or 1 for "yes"; default 1): 
```
- Based on the date string provided by the user, the code takes a guess of which directory it should look for data to reduce
- The code just appends the date string to the dir_root variable.
- Normally, you can accept the default directory. If you don't, the code will have you type out the full directory path
```
Should I redo and overwrite previous calibration calculations (Master bias, master dark, wavelength solution, etc) (y, Y, yes, Yes, YES, or 1 for "yes"; default 0):
```
- Do you want to overwrite previous calibration data (things like master Bias/master Dark/wavelength solution)?
- Normally you do not need to do this (default is to not overwrite)
- But, if you want to analyze data a little differently from last time, you should redetermine this calibration data:
- Examples:
  - You realize after the first reduction that some of the used darks had polluted illumination patterns, and want to redo the anlaysis without including them.
  - You took two types of wavelength calibration data, and want to see if the backup wavelength calibration method gives the same results as the default.
  - You only reduced half of the data the first time, and now have more of everything to analyze. So you will need to include more biases in the data calibration.
```
What prefix should I use to look for Bias images (default Bias): 
```
- What is the name of the bias (0s) exposures for this night?
- This will be whatever you put after the -p option when running the doPixisImaging.bash script on the OSELOTS control machine (see the Nightly Observing Sequence for more details).
```
Should I use Bias calibration data from the same night? (y, Y, yes, Yes, YES, or 1 for "yes"; default 1):
```
- Do you want to use bias data taken on this same night, or on a different night?
- Did you not take any biases for example?
- Generally for biases, you will want to use bias data taken on that night
- IF YOU ANSWER AFFIRMATIVE TO THIS QUESTION:
  - ```
  Enter the observation night (formatted as YYYY_MM_DD) with the Bias data you want to use:
```
  - Carefully enter the night of data you actually would like to use.
  - If the code cannot find a directory in the OSELOTS data directory archive with the specified name, it will ask you to type out the full path to the data you want to use.
The code will then print all images it finds in the data directory that start with the specified prefix, and ask:
```
Should I ignore any of these Bias images in the analysis? (y, Y, yes, Yes, YES, or 1 for "yes"; default 0):
```
- Are there any biases that the user does not want to use.
- Normally, there is no harm in using all bias images in the reduction.
- If the user does what to include a bias (answers 1 to the above question), the code will ask a followup:
- IF YOU ANSWER AFFIRMATIVE TO THIS QUESTION:
  - ```
  Enter the index (trailing number) of the next bad Bias image.  Just hit [RETURN] when done:
```
  - The user should enter, one-by-one, the indeces (trailing numbers, immediately before the .fits file extension) of the bias images they want to IGNORE
  - Type the first index to ignore, hit [RETURN], type the next index, and repeat.
  - When you have entered all indeces to be ignored, hit [RETURN] with nothing else entered.
- NOTE: The OSELOTS observing sequence can generate a lot of Bias images. If there are big chunks of data to be ignored, the user should consider deleting (on their local machine only!) the unwanted bias data
```
What prefix should I use to look for Dark images (default Dark):
```
- Same as above, except for Dark images (meaning images taken with a non-zero exposure time, with the lens cap on
- NOTE: In strict astronomy terms, these are not really darks, because the camera shutter is open. They are measurements of the on-camera illumination pattern that is NOT light coming from the optical fiber feed
```
Should I use Dark calibration data from the same night? (y, Y, yes, Yes, YES, or 1 for "yes"; default 1):
```
- Do you want to use Dark data taken on this same night, or on a different night?
- Did you not take any darks, for example, or are the darks not representative of the true illumination pattern at night (too much light in the dome, for example)?
- You are much more likely to want to use Darks from another night than Biases (Darks are not generally taken every night, as they require some setup)
- IF YOU ANSWER AFFIRMATIVE TO THIS QUESTION:
  - ```
  Enter the observation night (formatted as YYYY_MM_DD) with the Dark data you want to use:
```
  - Carefully enter the night of data you actually would like to use.
  - If the code cannot find a directory in the OSELOTS data directory archive with the specified name, it will ask you to type out the full path to the data you want to use.
```
Should I ignore any of these Dark images in the analysis? (y, Y, yes, Yes, YES, or 1 for "yes"; default 0):
```
- Same variety of questions as above, except with these 'Dark' images
- It is MORE likely that the user will want to ignore some Dark images, than Biases
  - If, for example, the dome lights were turned on or the sky was still bright, it may be better to ignore those darks.
- IF YOU ANSWER AFFIRMATIVE TO THIS QUESTION:
  - ```
  Enter the index (trailing number) of the next BAD Dark image.  Just hit [RETURN] when done:
```
  - The process for ignoring Darks is the same as that for ignoring Biases, above.
```
Should we use a wavelength solution determined on THIS night? (y, Y, yes, Yes, YES, or 1 for "yes"; default 1): 
```
- Do you want to determine the wavelength solution a new tonight?
  - Should be done whenever OSELOTS has been moved or disturbed
  - Usually a good idea to do first night of an observing run; not necessary after that
- Can only be done if you took wavelength calibration data (HG2 or KR1 data) on this night
- IF YOU ANSWER AFFIRMATIVE TO THIS QUESTION:
  - ```
  What prefix should I use to look for spectral calibration images (default HG2):
```
  - What was the prefix you entered when taking the data to be used as wavelength calibration data?
  - Generally either HG2 or KR1, the two sources of known wavelengths that we typically use to determine the wavelength solution
- ```
Enter the index (trailing number) of the next GOOD spectral calibration image.  Just hit [RETURN] when done: 
```
    - As the line explains, the code assumes most wavelength calibration data isn't useful (out of focus, saturated, ...).
    - Therefore, it asks you to specifically specify the trailing indeces of the calibration data that you want to use
    - The process of doing this is the same as specifying Biases/Darks that you want to ignore (see above).
- IF YOU ANSWER NEGATIVE TO THIS QUESTION:
  - What?

What prefix should I use to look for Sky images (default Sky):

What prefix did you use for on-sky images?

```
Should I ignore any of these Sky images in the analysis? (y, Y, yes, Yes, YES, or 1 for "yes"; default 0):
```
- Should any sky images be ignored?
- Although the default is 0, the answer is often 1. Some example reasons to ignore an image include:
  - The dome closed
  - The sky was too bright
  - The lens cap was left on
- IF YOU ANSWER AFFIRMATIVE TO THIS QUESTION:
  - Enter the index (trailing number) of the next bad Sky image. Just hit [RETURN] when done:
    - Specify the indeces of the sky images that you wish to ignore
    - Process for doing so is the same as for Bias and Darks, above.
- ```
Enter first sky image number when sky was dark (used to identify sky lines, over sun continuum); default [First index of sky images to analyze]):
```
  - What is the first sky image index that was taken when the sky was fully dark?
  - The actual listed default is the first index of the sky images that you elected to include
  - Important, because:
    - the code will measure properties of the spectrum from a stack of dark sky images
    - if images filled with bright sky continuum are left in, they will make picking out the profiles of the dark sky lines more difficult, reducing the quality of the reductions
  - Determining this generally requires looking at the data: where does the sky turn from a continuum to thickets of lines?
- ```
Enter last sky image number when sky was dark (used to identify sky lines, over sun continuum); default 283):
```
  - Same as above question, except what is the last image with a fully dark sky
  - The actual listed default is the last index of the sky images that you elected to include
- ```
Should I process different images with different focus positions as their own sets? (y, Y, yes, Yes, YES, or 1 for "yes"; default 1):
```
  - Sometimes, the observer might decide to move the focus position during the night.
  - In that case, should the code treat the two different focus positions as having different spectra?
  - If there are enough images in all spectral positions, the answer should generally be yes
The code will now start running. Your input will be needed again at the end, when the code is ready to save plots of the sky line profiles over the course of the night
```
Should I not plot any of these spectra? (y, Y, yes, Yes, YES, or 1 for "yes"; default 0):
```
- Are there any sky images that you don't want to include?
  - Maybe there is a bad image, maybe a line measurement is really bad in one image, etc.
- IF YOU ANSWER THIS QUESTION IN THE AFFIRMATIVE:
  - Which spectrum ID should I ignore next? (Just [RETURN] to finish identifying spectra to ignore):
    - Follow the same process for specifying which images to ignore, specifying the indeces of the sky images you don't want to plot spectra for
```
Should I not plot any of these lines? (y, Y, yes, Yes, YES, or 1 for "yes"; default 0): 
```
- Are there any spectral lines (identified by their line numbers, assigned in sequence of ascending wavelength) that you don't want to be measured?
  - Maybe they are too high you can't see the others, or maybe you don't trust they are real
  - Typically, you'll only know this after you have run the code once and decided you wanted to regenerate the line-changes-over-time plots
- IF YOU ANSWER THIS QUESTION IN THE AFFIRMATIVE:
  - Which line ID should I ignore next (ID starting with "L", NOT wavelength)? (Just [RETURN] to finish identifying lines to ignore):
    - Follow the same process for specifying which images to ignore, except you're specifying line numbers (ordered with an L, in sequence of ascending wavelength)
The night is processed. Congratulations.

Data Products

The code, if run successfully, produces the following data files:

Master bias data, consisting of:
- Bias.fits, a fits image containing the median 2d structure of the bias
  - The mean of each bias is subtracted out before combining
- Bias.txt, a text image listing the median bias level over time
Master dark data, consisting of Dark.fits, a fits image containing the median Dark data
If you had the code determine a new wavelength solution:
- The wavelength solution file
  - The first line is the 'anchor pixel' used to convert the 2D spectrum to a 1D spectrum
    - This is the row onto which the slightly-curve spectral lines are collapsed to form the 1D spectrum
    - Important because different anchor pixels would give slightly different column-to-wavelength solutions, since the same spectral line passes through different columns depending on the tow its in
  - OSELOTSWavelengthSolution.txt
- An image showing the matching between the identified lines and the reference spectral lines that the code was looking for
  - ```
  SpecSolution.pdf
```
- User should make sure that expected lines appear to fall on Gaussian peaks in measured spectrum
For each sky image:
- A fully processed version of the fits image
  - Denoted with the _proc ('processed') suffix
- A 1d spectrum text file
  - Denoted with the _spec ('spectrum') suffix
  - A three column table, showing the wavelength (in nm) associated with one (integrated) column, the inferred sky spectrum (in Ry / nm), and the total uncertainty in the inferred sky spectrum (also in Ry/nm
    - The uncertainty is inferred by running the same analysis on a patch of the image without a spectrum, and measuring the pixel-to-pixel variance
  - We also save a 1d spectrum image, plotting this data
- A 1d spectrum text file, including spectral steps
  - These data products are just like the 1d spectrum files above, except they include data on the data before applying the following spectral steps:
    - The sky spectrum before subtracting the continuum
    - The continuum-subtracted sky spectrum before applying the throughput
    - The final spectrum
  - and also the following calibration data:
    - The interpolated continuum
    - The throughput
  - These data are useful for checking steps in the analysis, to see if anything might have gone wrong