Analyzing Data

On this page, we describe how to quickly consolidate the data taken with the Aux Tel temperature array into a single, usable data file. 

This analysis, software, does not do much with the data; it simply takes the many data files taken over the night and merges them into a single usable data file. 

The code underlying this analysis is accessible at the Stubbs Group GitHub page: 
https://github.com/stubbslab/TempProbe 

Operation: 

1. Log onto the control computer: 
   1. You must have the appropriate VPN running on your computer (PSF1-LAS-UPD4-5787-config on Sasha's machine) 
   2. From terminal: 

   3. ssh stubbslab@auxtel-meerkat01.cp.lsst.org 

   4. (Enter usual stubbslab password)
2. Move to directory with data analysis code: 

   1. cd Documents/TempProbe/code/

3. Run the analysis code in python 3:
   1. python3 ConsolidateTemperatureVariance.py

4. Follow the code prompts: 

   1. Prompt: Enter the next date_str that will be added (just hit [RETURN] when you are done): 

      1. Enter the first date of the data you want to consolidate were taken on, formatted as YYYY_MM_DD 

      2. This process repeats until you RETURN with no string entered 

      3. For a night of data, these dates are generally the night you started observations and the following morning. 

      4. You can check what data were taken on which dates by checkin the: ~/Documents/TempProbe/data directory: 

      5. ls  ~/Documents/TempProbe/data

   2. Prompt:  Which is the first index from this directory that I should use (default {FIRST INDEX OF DATA IN THIS DIRECTORY}) :
      1. Enter the first index of the data on the shown night that you want to include in this consolidation 
      2. The data files are written in integer sequence, with the last number before the .csv suffix being the index.  
      3. You may not want to use all data from a date.  For example, data from the morning of one observing night can be saved in the same directory as data from evening of the following night 
      4. You can check where the cutoff appears to be by checking the time at which the files were written:
         1. ls -ltr ~/Documents/TempProbe/YYYY_MM_DD/*.csv 
   3. Prompt:  Which is the last index from this directory that I should use (default {LAST INDE OF DATA IN THIS DIRECTORY}) :
      1. Enter the last index of the data on the shown night that you want to include in this consolidation 
      2. See notes in point b. above. 
   4. Prompt: Enter indeces that I should ignore from this date, one at a time (just hit RETURN when done): 
      1. If you are aware of particular observations that you want to ignore, you can specify them here.  
      2. Code will ask you for indeces one at a time, until no string is returned 
   5. b., c., and d. repeat for each of the directories to be consolidated.
   6. Prompt: What is the root of the file to save (not including file type; default: TempVariance_YYYY_MM_DD): 
      1. What should the base of the files (plots and data files) that will be saved:
         1. Plots will look this this input, plus .pdf 
         2. Data files will look like this input, plus .txt
   7. Data are then saved to the ~/Documents/TempProbe/results directory 

Sample Results: 

Here are the results when we ran the above analysis to data taken on the night of 06/09/2022-06/10/2022: 

Bash script to concatenate data files: 
MakeVarFile.sh

Known issues and solutions: 

(Please add additional documented bugs, and fixes as they are found) 

Issue: 

Explanation:  

Solution: