This project is not yet finished. The project is outlined in this Overleaf document (editable at this link):
https://www.overleaf.com/6955685585cxpvzqqjypkz
The current task of the project is to determine two "nuisance" parameters in the fit (two of the parameters in Table 1): the binning radius, R_{bin}, and the grid density, R_{grid}. I think we should use five different values for R_{bin}: 100, 150, 200, 250, and 300 Mpc. These are chosen, as they span the general scale of superclusters/voids at the lower end and are the size of a PanStarrs medium deep field at the maximum redshift of z = 0.8 at the high end.
Ideally, R_{grid} would be arbitrarily small (we would do the fit in the entire sky). But that is not practically feasible. Instead, we should find the values of R_{grid} at which the SNe 3d peculiarity fits converge. In other words, we should run the full SNe 3 peculiarity effects at a range of R_{grid} values for each of the R_{bin} values, and determine at which point making R_{grid} better (smaller) ceases to matter. This is a somewhat lengthy process. We are partially done.
Once these convergence values are determined, we should run a series (~1000) of chains with data randomized. We keep track of the data in this sheet:
Green cells in the sheet are chains that have been completed.
Yellow cells are chains that have been executed on the cluster, but not completed.
Red cells are chains that we think should be run, but have not yet been started.
Icy blue cells are chains that we do not presently have any intention to run (they're "on ice")
Because the number of seed points can be quite large before we achieve convergence in R_{grid}, we divide the sky into a certain number of slices on the sky. Experimentation shows that ~180 slices is a good number. Therefore, for each of the five values of R_{bin}, we want to run ~ 10 X 180 = 1800 chains to find the value of convergence value of R_{grid} (that's the 10 in that product), and then an additional 100 X 180 = 18000 (and ideally 1000 x 180 = 180000) chains for the random bootstraps. That is a lot of chains. Keeping track of those takes some accounting. That's what the Google sheets are for.
Starting a new set of chains
Let's say we want to test the best fit value for R_{grid} = 24, 26, 28, and 30, and R_{bin} = 200, with the sky divided up into 180 slices.
We first need to generate the sbatch files for the chains. We can use the existing sbatch files as a basis. Let's use the existing R_{bin} = 250, R_{grid} = 30 file as our reference.
First, on the cluster, move into the appropriate directory (NOTE: for all the command-line code below, do not copy the $ sign - just copy and paste what follows it):
$ mv ~/stubbs/SNIsotropyProject/
Then, copy the reference file and (from the command line) change the appropriate line(s) in the file:
$ cp doSNIsotropyFit_Real_Grid32_Bin200_MinSN14_Z0p8_HemisphereBoth_Angle_1of180.slurm doSNIsotropyFit_Real_Grid30_Bin300_MinSN14_Z0p8_HemisphereBoth_Angle_1of180.slurm
$ sed -i "s/comoving_bin=200/comoving_bin=300/" doSNIsotropyFit_Real_Grid30_Bin300_MinSN14_Z0p8_HemisphereBoth_Angle_1of180.slurm
$ sed -i "s/comoving_grid=32/comoving_grid=30/" doSNIsotropyFit_Real_Grid30_Bin300_MinSN14_Z0p8_HemisphereBoth_Angle_1of180.slurm
Now let's copy that file for the other values of R_{grid} and update the appropriate line (do these one line at a time):
for i in 24 26 28
do
cp doSNIsotropyFit_Real_Grid30_Bin300_MinSN14_Z0p8_HemisphereBoth_Angle_1of180.slurm doSNIsotropyFit_Real_Grid"$i"_Bin300_MinSN14_Z0p8_HemisphereBoth_Angle_1of180.slurm
sed -i "s/comoving_grid=30/comoving_grid=$i/" doSNIsotropyFit_Real_Grid"$i"_Bin300_MinSN14_Z0p8_HemisphereBoth_Angle_1of180.slurm
doneWe now have the slurm files for the first of the 180 sky slices for each of the considered R_{grid} values. Now we need to generate the remaining 179 angle slice files for each. We'll do this with two nested bash for loops. This will generate 180 X 4 = 720 bash files (again, copy and paste one line at a time).
for i in 24 26 28 30
do
for j in {2..180}
do
cp doSNIsotropyFit_Real_Grid"$i"_Bin300_MinSN14_Z0p8_HemisphereBoth_Angle_1of180.slurm doSNIsotropyFit_Real_Grid"$i"_Bin300_MinSN14_Z0p8_HemisphereBoth_Angle_"$j"of180.slurm
sed -i "s/angle_slice=1/angle_slice=$j/" doSNIsotropyFit_Real_Grid"$i"_Bin300_MinSN14_Z0p8_HemisphereBoth_Angle_"$j"of180.slurm
echo Updated file: doSNIsotropyFit_Real_Grid"$i"_Bin300_MinSN14_Z0p8_HemisphereBoth_Angle_"$j"of180.slurm
done
done
And now we can repeat those for loops, executing these bash files:
for i in 24 26 28 30
do
for j in {2..180}
do
sbatch doSNIsotropyFit_Real_Grid"$i"_Bin300_MinSN14_Z0p8_HemisphereBoth_Angle_"$j"of180.slurm
done
done
We should now switch all squares on the rows with "Grid Size" = 24, 26, 28, and 30 of the 300 Mpc sheet from 1 to 180 from red to yellow.
Checking which chains are finished
All chains, once completed, get saved to the following directory:
$ cd ~/stubbs/SNIsotropyProject/SNeFieldFits/
To see if all 180 (or however many) chains for a particular set of parameters have been completed, you can perform an ls command on that directory, with an appropriate set of character substitutions. All final chain files share the following format:
OverdensityFitAAA_MinNSN_14_GridDens_BBB_BinSize_CCC_Hemisphere_both_RArange_DDDof180_RandEEE_fits.txt
where the three red capital letters represent a value specific to each file. The meaning of each set of three capital letters are as follows:
AAA - the sequence number of this set of chains, given these parameters (will be 1 for real chains, can be any number from 1 to whatever the max number of randomizations for the random bootstrapped chains)
BBB - the grid density, as an integer
CCC - the bin size, as an integer
DDD - the angle slice of the fit (will range from 1 to whatever the number of slices is, typically 180)
EEE - 0 if the chain is on real data, 1 if the chain is on random data
For example, if you want to check how many of the 180 sky slices for R_{grid} = 30 and R_{bin} = 300 are done, we move to the correct directory and then use the ls command:
$ ls OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_*of180_Rand0_fits.txt
Running this command gives:
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_100of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_1of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_101of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_21of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_102of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_22of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_103of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_23of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_104of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_24of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_105of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_25of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_106of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_26of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_107of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_27of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_108of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_28of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_109of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_29of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_10of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_2of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_110of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_30of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_111of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_31of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_112of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_32of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_113of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_33of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_114of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_34of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_115of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_35of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_116of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_36of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_117of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_37of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_118of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_38of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_119of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_39of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_11of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_3of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_120of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_40of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_121of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_41of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_122of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_42of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_123of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_43of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_124of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_44of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_125of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_45of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_126of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_46of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_127of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_47of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_128of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_48of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_129of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_49of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_12of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_4of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_130of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_50of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_131of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_51of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_132of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_52of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_133of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_53of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_134of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_54of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_135of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_55of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_136of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_56of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_137of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_57of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_138of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_58of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_139of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_59of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_13of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_5of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_140of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_60of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_141of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_61of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_142of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_62of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_143of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_63of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_144of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_64of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_145of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_65of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_146of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_66of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_147of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_67of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_148of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_68of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_149of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_69of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_14of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_6of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_150of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_70of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_151of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_71of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_152of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_72of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_153of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_73of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_154of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_74of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_155of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_75of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_156of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_76of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_157of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_77of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_158of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_78of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_159of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_79of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_15of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_7of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_160of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_80of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_161of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_81of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_162of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_82of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_163of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_83of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_164of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_84of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_165of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_85of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_166of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_86of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_167of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_87of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_168of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_88of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_169of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_89of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_16of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_8of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_170of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_90of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_171of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_91of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_172of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_92of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_173of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_93of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_174of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_94of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_175of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_95of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_176of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_96of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_177of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_97of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_178of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_98of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_179of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_99of180_Rand0_fits.txt
OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_180of180_Rand0_fits.txt OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_9of180_Rand0_fits.txt
Notably, chains 17, 18, 19, and 20 are still absent. So we should switch all squares on the row with "Grid Size" = 30 of the 200 Mpc sheet from 1 to 180 from yellow to green, except for 17, 18, 19, and 20. We'll have to wait for those chains to finish.
Note we can check an individual chain by listing that specific file. Say we want to see if angle slice 17 has finished yet. I would swap the * character in the row above for the specific 17:
$ ls OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_17of180_Rand0_fits.txt
ls: cannot access OverdensityFit1_MinNSN_14_GridDens_30_BinSize_200_Hemisphere_both_RArange_17of180_Rand0_fits.txt: No such file or directory
Getting a message that the file does not exist tells us we have still more time to wait.
Consolidating angle slices into a single data set
Say you've finished all chains for particular set of parameters (all cells from 1 to the number of angle slices are green). We now want to consolidate those into a single larger file. As I write this, I notice that all 180 angle slice chains for R_{grid} = 28 and 26, R_{bin} = 150 are done (green), but have not been consolidated. So we'll use those as an example.
Consolidating these chains again involves running a slurm script. Specifically, the
ConsolidateSNeOnSkyFits.slurm
script. The user will need to edit the following variables in this file:
rand
comoving_bin
fitter_ids
grid_densities
hemispheres_sets
angle_divs_set
The 'rand' variable should be either 0, if the chains being consolidated are from a real data set, or 1 if the chains being consolidated are from a bootstrapped data set
The 'comoving_bin' variable should be a single integer, whatever the size of the comoving bin is in the chains to be consolidated
The other 4 variables should actually be lists, formatted as [XX,YY,ZZ] (NOTE: the lack of spaces is essential!!!!)
The fitter_ids list should be whatever ids belong to the chains to be processed. For real data, this should always be [1]. For randomized data, this can be whatever the randomized fitter ids are. For example, [1,2,3,4,5]
The other three lists should be the same length, equal to the number of different R_{grid} variables you want to consolidate with this processing run.
The grid_densities variable should be the grid densities of each chain.
The hemispheres_sets variable should pretty much always be [both,both,both] , one both for each grid element
The angle_divs_set should be the number of slices into which the hemisphere is divided.
For example, to process the two data sets from we're interested in, I will set:
rand=1
comoving_bin=150
fitter_ids=[1]
grid_densities=[26,28]
hemispheres_sets=[both,both]
angle_divs_set=[180,180]
You can do so using vi:
vi ConsolidateSNeOnSkyFits.slurm
A useful manual for basic vi editing tips can be found here: https://www.ele.uri.edu/faculty/vetter/Other-stuff/vi/text_edit.html
The commands you'll mostly need are:
-Entering edit mode: type i, then you can type characters and delete them at your leisure.
-Saving your edits: Type : and then w
-Exiting vi once you've saved your edits: Type : and then q
-Exiting vi WITHOUT SAVING (so without typing : and w before hand): Type :, then q, then !
With those edits made, let's execute the command:
sbatch ConsolidateSNeOnSkyFits.slurm
Note the output file. When the batch job is done, open op the output file. The figures of merit (chi square improvement) are printed near the end of the output file.