Queued Service Observing with MegaPrime, WIRCam, and ESPaDOnS: Semester 2008A Report 09/03/2008

TABLE OF CONTENT

A - Introduction
B - General Comments
C - Global Statistics, Program Completeness, and Overheads
D - Agency Time Accounting
E - Conclusions

A - Introduction

The Queued Service Observing (QSO) Project is part of a larger ensemble of software components defining the New Observing Process (NOP) which includes NEO (acquisition software), Elixir (data analysis for MegaCam), `I`iwi (data analysis for WIRCam), Upena (data analysis for ESPaDOnS) and DADS (data archiving and distribution). The semester 2008A was the first semester in the history of QSO during which three instruments were offered in that mode: MegaPrime, WIRCam, and ESPaDOnS.

The main development for semester 2008A was of course the switch of ESPaDOnS to QSO mode after over 3 years of classical use. Thanks to careful planning and preparations, the transition went very smoothly. In addition, many new functionalities and improvements were incorporated to increase the quality and value of the data, such as automated telescope focus sequences, telescope focus model, automated Image Quality measurement, quick-look data analysis with reports of S/N.

B - General Comments

MegaPrime

The 2008A semester for MegaPrime started with seriously bad weather, but we were able to catch up and ended up with excellent completion rates.

1. Technically, the entire chain of operation, QSO --> NEO --> TCS, is efficient and robust. The time lost to the NOP chain is negligible. The system is quite reliable and efficient.
2. The QSO concept is sound. The possibility of preparing several queues covering a wide range of possible sky conditions (absorption and Image Quality) in advance of an observing night is essential for imaging in the visible, where seeing can change quite a lot, and very quickly. Thanks to the flexibility allowed by the QSO mode, a large fraction of the exposures were taken within constraints. The ensemble of QSO tools allows also the quick preparation of queues during an observing night for adaptation to variable conditions, or in case of unexpected overheads. The QSO mode also makes possible time constrained programs such as the CFHTLS. As usual, for 2008A, the global validation rate (validated/observed) for MegaPrime was excellent (92%).
3. QSO is well adapted for time constrained programs.The Phase 2 Tool allows the PIs to specify time constraints, even very restrictive ones. Time constrained programs complicate the planning and scheduling but CFHT provides tools for both PIs and Queue Coordinators.
4. Very variable seeing and non-photometric nights represent the worse sky conditions for the QSO mode with MegaPrime. Snapshot programs and regular programs requesting mediocre conditions (1" to 1.2") are used to cover those conditions. Fields requesting photometric sky conditions but originally done during non-photometric conditions are calibrated on perfect nights. SkyProbe and real-time measurements of the transparency are used to decide what observations should be undertaken.

WIRCam

The 2008A semester for WIRCam went very well, with some periods of bad weather, but not affecting observations as badly as for MegaPrime.

1. Technically, the entire chain of operation, QSO --> NEO --> TCS, is efficient and robust. The time lost to the NOP chain is very small. Certain operational modes specific to WIRCam, like nodding (target-sky-target...) and chip-to-chip dithering, have higher, unavoidable, overheads but some of them are charged directly to PIs during Phase 2.
2. The QSO concept is sound. As with MegaPrime, the possibility of preparing several queues covering a wide range of possible sky conditions (absorption and Image Quality) in advance of an observing night results in a very large fraction of the observations done within the specifications. For WIRCam, the sky background is more of a factor although its global variation through the night on Mauna Kea is fairly well known. Seeing is of course another important parameter but variations during the night in the near-IR are generally not as brutal as in the visible.
3. QSO is well adapted for time constrained programs.The Phase 2 Tool allows the PIs to specify time constraints. We can handle those easily if the weather is cooperative although the introduction of time constrained observations on a large-scale adds up definitive complexity in the scheduling process.
4. Non-photometric nights represent the worse sky conditions for the QSO mode with WIRCam. An important difficulty on near-IR astronomy is the removal of the sky background. Non-photometric conditions make that operation a more difficult one. Nodding for instance cannot be done. The availability of SkyProbe and real-time measurements of the transparency is extremely valuable and regularly used do decide what observations should be undertaken. Also, the real-time analysis through `I`iwi provides a direct estimate of the extinction through the 2MASS catalog, helping even more the observing process.

ESPaDOnS

The 2008A semester with ESPaDOnS was a great success. The only major issue was a software bug which caused the instrument to be incorrectly configured for the 4th exposure of each polarimetric set of 4 (spectroscopic observations were not affected). We re-observed all targets affected (as much as we could), at no charge to the PIs, and reduced the incomplete sets of data with only 2 exposures (again, at no "cost" to the PIs). CFHT also implemented improvements to the guiding algorithm, and to the telescope focussing procedure. For the first 2-3 nights of an ESPaDOnS run, Service Observers perform a 2-step focus sequence using exposures from the ESPaDOnS guider. The data gathered during those sequences are used to update a telescope focus model which is then used for the rest of the run, without having to refocus on each target. An automatic Image Quality measurement routine has also been developed to get a measurement of the seeing on each target.

1. Technically, the entire chain of operation, QSO --> NEO --> TCS, is efficient and robust. The time lost to the NOP chain is small, but there is some room for improvements. Most of the overheads come from acquiring (finding) the targets, performing full telescope focus sequences for the first 1-3 nights of a run, measuring the Image Quality, and initiating the guiding.
2. The QSO concept is sound. For ESPaDOnS, and in contrast to our imagers, seeing and extinction are much less of an issue and do not factor much in the preparation of the queues. Queue Coordinators usually prepare one or at most 2 queues per night. The advantage of the QSO mode comes from the ability to schedule observations exactly when they are needed.
3. QSO is well adapted for time constrained programs. ESPaDOnS observations are characterized by a high demand for time constrained observations and monitoring requirements: about half of the programs had special requests for timing. The queues are usually prepared by taking first into account programs with strict time windows. It was not unusual to juggle a program with a target to observe every night, a second program with a target to observe 2-3 times a night with 1-hr gaps between Observing Groups, a third program with a target to observe every 2 or 3 or 4 nights, and 2 programs to execute within a certain time window from one another. There are also programs that necessitates continuous blocks of 4 to 8 hours during the same night. The PH2 tool allows PIs to specify all sorts of time constraints, and add any comment to help the QSO Team select appropriate programs.
4. ESPaDOnS can deal with non-photometric nights and bad Image Quality Except for very faint targets (fainter than about 13) which can be difficult to find and center with cloudy conditions or highly degraded seeing, most observations can be carried under a very wide range of sky conditions. Extinction and bad seeing reduce the amount of flux getting into the instrument, but the Service Observers compensate by repeating Observing Groups (at no cost to the PIs) to recover some of the lost flux.

C - Global Statistics, Program Completeness, and Overheads

(1) Global Statistics

The following table presents some general numbers regarding the queue observations for 2008A (C, F, H, L agency for MegaPrime, and T, D-time, excluding snapshot programs unless noted otherwise). Note: to convert from number of hours to number of nights, a factor of 9.5 hours per night was used.

Notes concerning MegaPrime

• The MegaPrime time lost is mainly due to weather and not technical problems. MegaPrime was the instrument most affected by bad weather.
• The time lost due to engineering and technical problems is quite low for 2008A. This is due to a series of efforts which focussed on solving recurring fiber and Slink card issues.
• Programs not started at all were either C programs (which exist to overfill the queue), Snapshot programs, or a very low rank B program.
• While A+B programs were completed at 92%, the completion rate of A programs is over 96%, which is excellent.
• The global validation rate for A+B+C+S programs (validated/observed) is very good at ~86%. Exposures which cannot validated were taken under horrible sky conditions, as the QSO Team often tries to observe even under bad sky conditions in the hope that the weather could improve. The Validation Efficiency is higher (92%) because it uses the validated/requested number of hours.
• The total number of hours validated during the semester is ~390hrs, which gives 390/79 = almost 5hrs per night of observing. This is 0.5hrs below the objective of 5.5hrs. The main factor explaining this shortcoming for 2008A is the weather.

Notes concerning WIRCam

• The weather was obviously better during WIRCam runs compared to MegaPrime ones. This is reflected in the higher number of completed programs, the higher completion rate (A programs were completed at 100%!), and the higher validation rate.
• The total number of hours validated during the semester is ~287hrs, which gives 287/50 = 5.75hrs per night of observing, very close to the 6.0hrs/night goal.

Notes concerning ESPaDOnS

This was the first semester with ESPaDOnS in QSO mode, and significant differences were found in terms of queue preparation, handling of priorities (grades/ranks), and particular scheduling difficulties.

• The time lost to weather was mostly due to fog. Cloudy conditions usually do not prevent observations, as would be the case for the two imagers MegaPrime and WIRCam. ESPaDOnS is much more tolerant of bad sky conditions!
• The time lost to engineering during the first run was mostly due to on-sky tests of the whole NOP chain. The fact that only one night (over the whole semester) was needed to perform the last checks and test new functionalities shows how well the switch to QSO had been prepared and pre-tested.
• The time lost during the second run was mostly due to the (totally unrelated) failure of telescope encoders. The engineering time spent during the last run was used to take data in order to refine our telescope pointing model.
• The first 5 nights were affected by a software bug which prevented the polarimeter from being configured correctly for each 4th exposure of a polarimetric sequence (6.5hrs of spectroscopic observations were not affected). About 8.25hrs of polarimetric data taken with no or less than 0.5mag extinction were either lost (because the window of opportunity had passed) or had to be repeated later during the run (only 1/4th of the 8.25hrs are taken with an incorrect instrument configuration, but even just 1 bad exposure makes the whole polarimetric sequence unusable). Another 9hrs were affected in a similar way, but were taken under poorer sky conditions, with at least 1 magnitude of extinction. The QSO Team basically retook those 17hrs of observations (at no charge to the PIs).
• All ESPaDOnS programs were started except one, which had a low rank, needed a considerable amount of telescope time (30hrs), and was in direct conflict with highly ranked programs.
• The low number of completed programs (only 14 of the 24 started programs) and the low completion rate in general (82% for A+B programs) is due to the presence of many time-constrained programs (about half of all programs): when an opportunity to observe a target is gone (usually due to weather for 2008A) and the scheduled time window has passed, there is no way to complete a program as initially prepared. It will probably have to be accepted that the completion of ESPaDOnS programs will be lower when compared to MegaPrime and WIRCam. It should be noted, however, that despite this difficult characteristic of ESPaDOnS programs, 6 of the 9 A programs were completed at 100%, and the completion of all A programs is still excellent, at 95%. The B programs are those which suffer the most (as designed for QSO!).
• The total number of hours validated during the semester is ~248hrs, which gives 248/41 = a little over 6 hrs per night of observing, which is still quite short of our goal of 7.5hrs per night. However, engineering nights are included in that 41 night total. Removing the 3 nights scheduled for Engineering results in an average of 6.5 hrs per night, which includes the effects of nights lost to fog and overheads. ESPaDOnS produces quite a lot of data when the weather is good and observations are conducted efficiently: in July, it was not unusual to get about 8h45m of integration + readout time per night! Given that a July night lasts about 9h45m between the 8degree twilights, this is quite good: about 5min are spent per target to slew the telescope, find the target, focus the telescope, get an IQ measurement, and start the guiding. These numbers agree with the overheads estimates given below.

(2) Program Completeness

MegaPrime

The figure below presents the completion level for all MegaPrime programs in 2008A, according to their grade:

Remarks:

• The global completion level with MegaPrime for A programs is ~96% while B programs were done at ~83% . These values are excellent.
• The completion rate for some C programs is relatively high. This shows that C programs are very useful for QSO if they require modest conditions and when the targets are not located in RA ranges too populated by other highly ranked programs.

WIRCam

The figure below presents the completion level for all WIRCam programs in 2008A, according to their grade:

Remarks:

• The global completion level for A+B programs is very high (97%), and all A programs were completed. Two C programs were also completed at 100%, while the other 2 C programs were not started. This is excellent!

ESPaDOnS

The figure below presents the completion level for all ESPaDOnS programs in 2008A, according to their grade:

Remarks:

• The global completion level for A+B programs is low (82%) compared to the other 2 instruments, but the A programs were completed at 95%, which is excellent.
• As mentioned earlier, a low rank B program was not started because of a direct conflicts with higher ranked A programs.
• The two C programs were completed at 100%, which clearly shows that C programs are needed with ESPaDOnS, mostly to fill ranges of RA underused by A and B programs.

(3) Overheads

MegaPrime

The following table presents the main operational overheads (that is, other than readout time of the mosaic) with MegaPrime. These numbers have not changed, and are given as a reference. Overheads are highly variable during a given night depending on the conditions, complexity of science programs, etc. The table below shows that overheads take a maximum of ~35min per night. A short summer night lasts about 9 hours with MegaPrime, so overheads take less than 10%. The number originally expected was 10-15%.

Notes:

• Overheads to change filters are important, but this is done in parallel during readout or while the telescope is moving. Therefore, there isn't always an overhead due to a change of filter. The global overheads also depends strongly on the number of standard stars observed for a given night and also if switching from a queue to another is necessary (since overheads due to filter change are minimized within a specific queue).
• Focus sequences have been almost completely removed from operations. The auto-focus model has been available for a while and significantly increases the time spent observing instead of focussing. A few focus sequences are taken during the first nights of a run to confirm the zero points of the model, but after the model has been updated, no more time is spent on focussing
• Overheads due to dome rotation are minimized as much as possible within a specific queue. Dome rotation is now optimized and cannot be made faster.
• Guide star acquisition is fully automated and except from some rare problematic acquisitions, it works really well. Acquisition tends to take longer when the seeing is bad or cirrus are present. Programs with frequent guide star acquisition with short exposure strategy increase the global overheads. The main overhead related to the guide star acquisition has been reduced dramatically in 2005A by accelerating the probe motion. Dithering patterns offsets for instance are now completely hidden in the readout time, which was not the case in the past.
• Overheads for calibrations (standard stars and Q98 short exposures for photometric purposes) are not included in this table. In 2008A, we observed standard star fields twice per photometric night, using only 1-2 filter per night. Only 7.8hrs (including readout time) were spent on standard fields for the whole semester, or 6min on average per night.

WIRCam

The following table presents the main operational overheads (that is, other than readout time of the mosaic) with WIRCam. Overheads are highly variable during a given night depending on the conditions, complexity of science programs, etc. The table below shows that overheads take a maximum of ~25min per night. A short summer night lasts about 9.5 hours with WIRCam, so overheads take around 5%, which is quite low.

ESPaDOnS

The following table presents the main operational overheads (that is, other than readout time of the mosaic) with ESPaDOnS. Overheads during a given night depend mostly on the number of targets and the need to change Observing Mode or not. The table below shows that overheads can take up to 1hr per night, most of that coming from the acquisition stage (pointing the telescope, finding the target, centering the target, starting the guiding). A short summer night lasts about 9.5 hours with ESPaDOnS, so overheads take around 10%.

Notes:

• The change from one CCD Readout Mode to another takes no time; the change from one Observing Mode to another, if done during the night, is accompanied by a spectrograph focus and takes 7-8 minutes (mostly because the spectrograph focus requires 4 exposures). During 2008A, the 3 Observing Modes were requested, and change from one mode to another during a night was inevitable. This overhead will be zero for semester with only one Observing Mode requested.
• When only 1 Observing Mode change is required, no calibrations need to be taken during the night; the reduction uses calibrations taken during the late afternoon or at dawn. If 2 or more Observing Mode changes are required, calibrations must be taken during the night, and can take up to 20-25 minutes of sky time. This is very costly and the QSO Team tries to avoid these cases as much as possible. During this semester, we only had 3 nights with onerous changes of Observing Mode during the night.
• During the first QSO run with ESPaDOnS, a lot of time was spent taking telescope focus sequences, which give an objective calculation of the optimum telescope focus position (a significant improvement over the focus-by-eye done when ESPaDOnS was used in Classical mode!). Those sequences were used to setup a focus model, which was used during the following 2 runs. The established routine is to now take telescope focus sequences during the first 1-3 nights of a run, on 6-12 stars per night. As the model gets better, even less focus sequences will be required.
• After the telescope is in focus, a few seconds (30-60s) are also taken to calculate the Image Quality on each target, using again an automated sequence. A couple a minutes are spent making sure the right target is picked, using Finding Charts provided by PIs.

D - Agency Time Accounting

(1) Global Accounting

Balancing of the telescope time between the different Agencies is another constraint in the selection of the programs used to build the queues. After this first semester with ESPaDOnS in QSO mode, it was found that this is much more difficult to do with ESPaDOnS than with the other 2 instruments, because a lot of ESPaDOnS programs request strict time constraints or have narrow windows of execution, which takes precedence over Agency Balancing. The queues are basically done according to the requested time constraints and try to follow the programs' ranks as much as possible. The agency balance is whatever it ends up being at the end of the semester. This being said, the agency balance for ESPaDOnS was quite acceptable for 2008A.

MegaPrime

The figure below presents the Agency time accounting for 2008A.The top panel presents the relative fraction allocated by the different agencies (programs A + B), according to the total I-time allocated from the Phase 2 database. The bottom panel represents the fraction of observations validated (programs A+B+C) for the different Agencies, that is, [Total I-Time Validated for a given Agency]/[Total I-Time Validated]. As showed in the plots, CFHTLS is the agency lagging behind, with less validated I-Time than allocated. The significant fraction of time lost to bad weather (~30%) contributed to making the agency balance difficult to obtain.

WIRCam

The figure below presents the Agency time accounting for 2008A.The top panel presents the relative fraction allocated by the different agencies (programs A + B), according to the total I-time allocated from the Phase 2 database. The bottom panel represents the fraction of observations validated (programs A+B+C) for the different Agencies, that is, [Total I-Time Validated for a given Agency]/[Total I-Time Validated]. As showed in the plots, the Taiwanese agency is well ahead while CNRS is slightly behind.

ESPaDOnS

The figure below presents the Agency time accounting for 2008A.The top panel presents the relative fraction allocated by the different agencies (programs A + B), according to the total I-time allocated from the Phase 2 database. The bottom panel represents the fraction of observations validated (programs A+B+C) for the different Agencies, that is, [Total I-Time Validated for a given Agency]/[Total I-Time Validated]. As showed in the plots, the balance is good, with the French agency a little ahead, the Taiwanese ahead, and Canada behind (mostly because of the 30-hr program that could not be started).

(2) CFHTLS Accounting

CFHTLS occupies a large fraction of the I-time allocated for QSO for MegaPrime. The following figures show the time accounting for the different CFHTLS components for 2008A (L01 and L04 are the Deep Survey, L02 and L05 are the Wide Survey, L03 is the Very Wide Survey, and L99 is the photometric grid for the Wide Survey):

Fraction Requested

Fraction Validated

After grouping together the programs for each component of the CFHTLS, the numbers confirm what is seen in the plots: the photometric grid is well behind with respect to the surveys themselves. The global fractions are presented in the following table:

E - Conclusions

MegaPrime

The 11th semester with MegaPrime in QSO mode was overall very good. Bad weather again hit MegaPrime the hardest, but overheads have been decreased, the completion of A and B programs and agency balance are very good.

WIRCam

The 6th semester with WIRCam in QSO mode was excellent. The programs completion and efficiency are the highest of the 3 instruments.

ESPaDOnS

The first semester with ESPaDOnS in QSO mode, after 3 years of Classical use, went very well. The transition went very smoothly and a minimum of time was lost to engineering. New functionalities, such as automated telescope focus sequences, telescope focus model, and automated Image Quality measurement, were successfully developed and implemented, insuring the best data quality possible. Preparing queues with ESPaDOnS turned out to be completely different than preparing queues for our two imagers, because (1) ESPaDOnS does not care as much about Image Quality and amount of extinction, and (2) many programs have requests for time constraints.