https://summit-lsp.lsst.codes/chronograf/sources/1/dashboards/65?refresh=Paused&lower=now%28%29%20-%2015m

2024 May 13

We looked at images (307-316) and we needed threshold of 500, box size of 100 to work.

For images 317-416:

Elana code is finding all four beams and nothing else in every image so far, and is able to centroid the beams!! Exciting. we are taking more data. One of the PSFs is garbage, but we will try convolving and see where that gets us.

Questions:

• what in the heck was happening with the focus and PSFs earlier?
  • ATMCS cRIO needed resetting? down
  • maybe primary not supported on pneumatics
    • still not understanding this - with no pneumatics maybe motion, but also short time scale
• next steps:
  • measure/calculate/compare the distances between beam centers
  • take as much data as humanly possible (in progress)
  • separate notebooks for quick look and full blown analysis (easy!)
  • making whole system remote
  • do while observing (depends on clouds)
• power law convolving with data analysis on fourth peak?
• why is PSF of fourth beam bad (beam closest to beam expander)? 
  • meghan hypothesis that maybe has something to do with light being clipped by edge of mirror
    
    

difference between dome azimuth and telescope azimuth

→ what would it take to re-acquire from scratch (meaning 

sine convention

check rotator angle post resets, plot as a function of time (look for glitches)

plot flux!!

• frame of the dome analysis
• sine conventions piezos
• plot flux

consistency with relative flux: dewar window and dome rattling (and maybe some beam blocking)

checklist:

• startup
• align
• find beams
• drive beams to desired generation
• second generation of coolness

System is basically working. Four spots on LATISS sensor with around 3 arcsec FWHM. 

What PSF do we expect? Beam is about 40mm in diameter. Diffraction limit would be 1.2 * lambda/D ~ 0.5E-6/40E-3 = 3 arcsec. 
We're close to that now. Clear diffraction rings around main sources 20240510 seq 52: 

Initial Jupyter analysis notebook is in github repository https://github.com/stubbslab/MOSS 

Here is azimuth configuration that placed beam onto detector: 

Assuming our azimuth convention is zero at N and increasing to the East, then dome slit is pointing roughly North and telescope roughly East. 
Compass check in dome has slit pointing to az~45 and telescope at 82 deg. So telescope seems OK but dome zero in azimuth seems way off. 

Dome Closed Initial Differential Motion Analysis (images 317-327 from 20240513)

Dome Open Analysis (images 1757-1767 from 20240513)

Dome Open II (images 2525-2535 from 20240513)

Discussion with Chris, Elana, Chuck (significant scope creep)

• Chuck: stratification in the dome, eddy currents, then an initial flushing when the dome is open, and then return to equilibrium after the dome open for a while (for LHS plots with elongation)
• Chris: magnitude of separation between pairs does matter DIMSUM - correlation length
• Misleading by centroiding errors?
• turn into lockin if you pulse the laser at 20 Hz = strobe coherent oscillation
  • integrated strobed camera
  • camera (readout time) as low pass filter after mixer
    • high freq components will be superimposed on low frequencies?
    • nyquist sampling frequency wraps signals
    • base band = things you can nyquist sample at the frame rate of the camera
    • drive with square wave or pulse?
      • no intensity issues - mean intensity is always the same over frequency
  • will work on frequencies up to twice the pulse duration → characterize the turbulence up to a kHz
  • speed of sweeping is limited by camera frame rate
    • exposure time * frequency repetitions on a single camera readout
    • assume frozen turbulence per pulse 
    • chris: this only allows searching for coherent components
      • if q (>= exposure time*frequency)  high enough that it's the same phase when strobed, get rid of differential motion
      • move away from that frequency - beat frequency
• what experiments can we do to inform the next generation of this device
• shutter open/closed synchronization?
  • listen to the shutter with a microphone
  • Chuck has an electrical interrupt solution
  • supposed to be a published event for both auxtel and main dome

Todos:

• block beams
  • repoint at same angle
  • block one at a time
• remote operations
• shorten pulse to surpress dome movement (chuck thinks this is miniscule)
• dome testing
• frame of the dome analysis
• square the axes on the first plots - and test that the line is at the elevation angle 
  • this would be a good confirmation - centroiding errors cannot know the dome angle
• check that no differential image motion with 10s image?
  • measure standard deviation as a function of pulse duration - coherence time of turbulence in the dome; change in characteristic time with dome open, dome closed? 
• align spots for stuttered imaging mode at current frame rate

Pulse Length/ND Filter Discussion

Chris ran reductions blindly without looking at plots. We should all reference observing logs to get a feel for PSFs or plot a few before proceeding with full analysis on plots.

Notes

• short pulses seem completely dominated by some artifact
• chris guess: laser diode turns on, heats up the second is energized
  • some thermal transient that makes emitted beam profile different for super short pulses
  • for short pulses, beam coming out of laser diode has more divergence (we don't seem to have collimated beam in this case)
  • beam expander collimation for beam on longer time scale
• we should go to the shortest beam possible that gives us good PSFs

Todos

• relevant PSFs as a function of pulse duration - plot next to each other
• go to OD 1.5, take more .1ms images to make sure PSFs/FWHM still good and we get pixel max values of 10x

Data Reduction Standards

Reduction is saved under date folder in all_peaks_df in data in MOSS repository.

There are columns in the data frame with dome status, OD filter, convolution sigma, box size, and threshold, as well as date and image numbers.

5.14 Analysis Notes

Image numbers in chart, 25 from each. We are planning to compare these. Green when all_peaks dataframe has been saved. Using conv_sigma = 35.

5.15 Analysis

Ran through images on MOSS - getting standard deviation of 400 in y direction for 3970 stack. 

• Running more images now to see if this is just the stack I pulled
• had to switch to sorting in y in the function
• Elana suggested outlier rejection - discuss further

Interpretation

Information about rot_pa in the header: https://ts-observatory-control.lsst.io/user-guide/tcs-user-guide-generic.html#rotator-position-and-sky-position-angle

MOSS On-Sky Observation Plans

Goal: without moving the dome, alternate between taking g-band on-sky images at a wide range of airmasses and 

1. run ATpneumatics_checkout
2. open dome: auxtel/atdome/open_dome.py
3. open mirror covers (component: ATPneumatics, cmd: openM1Cover)
4. enable corrections:
   1. 
5. With run command, disable dome following.
   1. 
6. run correct pointing (put range for target there)
7. try to run latiss_wep_align to focus telescope - may not work, may have to do manual focus sweep
8. NOTE: FIND CORRECT AZ POINTING FOR THE DOME
9. Find targets of mag 12 +/- 1 at a large range of airmasses/elevations and take images of those without moving dome
   1. 
   2. take_image_latiss as engtest → with g band in filter wheel, reason is on-sky, probably
10. stop tracking
    1. 
11. point to MOSS and take a series of images there
    1. 
    2. take_image_latiss

Summer meeting notes:

• Merlins thing basically works
• Juan Lin’s does much better ISR
• —> take their ISR, then do a peak finding and source fitting for both getting moments and centroiding using merlin’s method and gals
• Now, go through the data, do rotation so that we are in x and y, and seeing what we see (rotate the centroid coordinates after)
• preops notebook in GIT!!!!
• Improperly masked bad columns? - ISR mostly has worked
• Not perfect - going to have to check for outliers, etc.
• Running through the full 9000 images has to be done
• Elana knows relationship between rotation az/el and we will matrix and put in the sign

Notes 9.22

from this plot of data from two images, it seems like the sourcing pair that is split is the sources at x distance of ~0 

Looking at the plot, this is probably the bottom two sources (which makes sense, as the very bottom source is the one with the bad centroiding).

Now going to try finding why they split.

Why does it seem like we are still plotting the dataframe with data that does not necessarily have all four sources? Are we plotting over points here?

Split the plot into plotting by the source pair (0-5); it seems that

a) sources are not being paired consistently, so we have a few groups that are split 1/2 and 1/2 

b) the smaller two groups are actually from the same pair of beams, and the distance between them in y is changing dramatically

Now, we want to find which physical beams are in pair 3. From the dataframe it looks like pair 3 is composed of source_id_1 = 1 and source_id_2 = 2.

• Are we misidentifying one of the sources sometimes, or are we actually seeing that much motion in the same pair?

Which x and y coordinates do these source_id values correspond to?

Plotting our four groups of peaks gives me this, which does look like we are plotting the same shape as the images I have that look like the above. However, maybe it's the COM plotting that is being difficult; when I plot grouping by the positions relative to the COM in x and y (as the pairwise distances function does), I get weird chunks in my groupings that don't conform to the expected peak positions. (third image is to illustrate that we have the proper shape by making the x and y scales even, although it's upside down)

Why would the center of mass function be splitting up these groups? The code for the COM groups is:

'''

def calculate_com(group):
# Compute Center of Mass (COM)
com_x = group['centroid_x'].mean()
com_y = group['centroid_y'].mean()

# Add COM to the DataFrame
group['COM_x'] = com_x
group['COM_y'] = com_y

# Calculate position relative to COM
group['pos_rel_COM_x'] = group['centroid_x'] - com_x
group['pos_rel_COM_y'] = group['centroid_y'] - com_y

return group

'''

It is applied to the dataframe, and then new columns are added relating to COM. I think it must be the .mean() call from the calculate_com function, because the other major operation is just subtraction and should not throw relative spacing off (should just shift it). This comes into play in our differential_motion_analysis code because the pairwise distances calculates distance using the COM values!

Writing a new function nonCOM_pairwise_distances to see if this is truly the problem

New function does not seem to have changed anything even though I wrote in that the 'centroid_x' and 'centroid_y' columns should be used as opposed to the COM columns.

Think it is swapping the peak id of two values, so I need to re-sort them in the dataframe (add a new column of source id!).

Once sorted in y, the source assignment improves dramatically (this is May 15 images 4200-4350):

However, we can see we still have some points in a different location (some common-mode motion maybe judging off of the left plot). We ca probably set a filter on the x,y positions to make these plots really nice and have no contamination between beams (though I wonder how the contamination is happening when if it's common-mode motion, the peaks should all be mapped the same relative to one another...).

I want to try some other data as well, but this functionality is now built in!

We were using images that only had three correctly centroided sources, and filtering this out (if centroid_x or centroid_y are nans, get rid of the row), we have a normal looking image of the peaks again!

Running through the rest of the code here, we have:

Woohoo! This was images 4200-4350 on May 15. Time to do more images! And then we can curve_fit everthing...

Starting with comparison of dome-open, dome-closed on May 15 night.

9.28

Comparing dome open vs dome closed data on may 15

Dome open = 4200-4500

Dome closed = 2898-3825

DOME CLOSED

It does look like there is some drift in x separation initially here for the farthest apart pair...

DOME OPEN

Notes from meeting with Elana:

• get angle from butler parameters (check that it remained constant between shots, check that it scales with elevation angle) and then using a rotation matrix, rotate the values in the dataframe
• we expected the point clusters to be more condensed in the COM frame because it gets rid of the common-mode motion between beams
• want to see if seeing improves over the course of the final night when we had the dome open for an extended period of time
  • re-check on-sky data to see if anything can come of comparison
• compare over different pulse lengths / dome open percentages from May 14 data
• with the dome closed plot above where there's a shift in source x separation:
  • see if this is true for other pairs of beams
  • temperature comparison with in-dome sensor

Temperature querying:

temp_query_str_new = f'''
SELECT mean("temperatureItem0") AS "mean_temperatureItem0"
FROM "efd"."autogen"."lsst.sal.ESS.temperature"
WHERE time > '{start_time.isot}Z' and time <= '{end_time.isot}Z' AND "sensorName" = 'AuxTel-ESS02'
'''

Found query in chronograf to get the average time over each camera exposure!

plot of temperature for the above image sequence:

Added to same plot, met with Chris and Elana. Powerpoint attached below:

10.27

Talked with Elana last week about usability of the images with different pulse durations. Had a lot of images that look quite bad, and Elana agrees that even a different centroiding algorithm might not be the most effective way to go. 

Now want to:

• plot a window of anemometer data against the movement in x and y that was worrying with the temperature data previously
• figure out how the total standard deviation column is calculated such that it is less than the y in some cases
• compare in temperature for multiple sets of beam pairs
• change axis labels to aximuth and elevation after angle changes
  • more clearly split out in the code where these angle changes are
• for the oscillatory motion over time, figure out if we fourier transform, is there a power spectrum
• see if the temperature is increasing or decreasing over the course of a night, and if there is a corresponding change in beam separation for individual pairs
  • if there is a trend, check if it is consistent between nights
• plot standard deviation of anemometer temperature against differential image motion std. dev (dome seeing) to check if they are correlated
• analytic assessment of centroiding uncertainty: variance scales like sigma^2/flux, so standard deviation is sqrt of that (moved)

Documentation update 11.13

To-do:

• sonic anemometer analytic correlation
• figure out what episodic temperature jumps are due to
• correlation plots with delta x and delta y
• centroiding assessment (from above)
  • plot these in the rotated frame with ROTPA
  • figure out if there is also a correlation with temperature here
    

Presentation from  11.11 Notes:

• We have hardware that seems to work with the slight issue that dome jitter contributes problems
• Broadly speaking, rms increases with image motion between pairs
• The sonic anemometer data seems to bear no significant relation to moss data
  • Chris trusts MOSS more as it is still a more direct measurement of what we actually care about
• Episodic jumps between pairs with opening and closing between pairs
• We are not seeing systematic focal length shifts with temperature which is great!
• Image motion in x in y correlation - work in progress

Delta x Delta y correlation plots (point versions of lissajous plots from presentation)

10 Images

50 Images

300 Images

We definitely see many non-negligible R^2 values here, and I think they would be even more significant with the reduction of some outliers.

Slides 11/25 with notes from group meeting

Figure updates with rotation - December

To-do:

• Bootstrap data to get error bars and re-plot with error bars
  • use 95% CI
• Jacknife data too
• Split out data into 10 groups of 100 points and re-plot for each
• Bootstrap data for time-sequenced images
• Add data points at 0,0
• Switch graph labels to azimuth and elevation
• calculate centroiding error (FWHM/(S/N))
• make sure constrain scales on vertical axes of six plots to be identical

For some reason there is a factor of 10 difference in the dist_x and dist_y values that I bootstrapped vs. the ones in the other dataframe. Why? The error bars should at least include the points already calculated from my dataset...

So I'm thinking that was because I bootstrapped all pairs separately so the variance was extremely high. I also now thinking about it more think that it makes the most sense to bootstrap by image number.

The number of peaks I was getting in the dataframe was different for the bootstrap because I hadn't yet filtered for images with four peaks. Now, the bootstrap is better!