Differences between revisions 91 and 92
Revision 91 as of 2014-03-11 09:28:57
Size: 14934
Editor: obelix
Comment:
Revision 92 as of 2014-11-11 08:05:24
Size: 21769
Editor: obelix
Comment:
Deletions are marked like this. Additions are marked like this.
Line 2: Line 2:
== v1.3.6 ==
''(07.11.2014, by ESO)''
 * Recipes
    - kmos_illumination is failing on a single data set on detector 3.
      The reason is an unfortunate slitlets sanity check that fails on this
      particular data. The check is still considered good and the
      pipeline remains unchanged
      (https://jira.eso.org/browse/PIPE-5387)
    - kmo_sky_tweak is failing on a single data set with endless
      errors. The error handling is improved.
      The checks to detect NANs have been improved - Problem solved.
      The recipe is renamed kmos_sky_tweak
      (https://jira.eso.org/browse/PIPE-5194)
    - Extracted spectra contain NaN values when extracting with a mask
        (https://jira.eso.org/browse/PIPE-4997)

 * Calibration Scripts
     - easySPARK_calibration fails if the OB crashed, although all the
      data ave available
      This ticket is adressing the easySPARK scripts issues in
      general. A deep analysis of the needs and the reported issues (4
      cases of emergency support for Paranal Science operations
      occured in 2014) lead to a full re-write.
      The new algorithm for files automatic selection / grouping and
      recipes execution has been discussed and agreed upon with
      Paranal Science Operations during the October 2014 visit.
      The easyPLUG (graphical tool - 4000 l. of perl) and the easySPARK
      scripts (3000 l. of shell) software are not delivered any more
      from this version on.
      The calibration files generation needed in Paranal is exclusively
      covered by the new script kmos_calib.py (500 l. of python)
        (https://jira.eso.org/browse/PIPE-5076)

== v1.3.5 ==
''(08.09.2014, by ESO)''
 * Recipes
    - kmos_flat made more robust - Each change in the arm configuration
        has for consequence to make the slitlet detection fail.
        The threshold used is dynamically reduced in case of failure. This
        solves the 6 reported failing cases, and hopefully will make the
        recipe more robust for the next configuration change.
        (https://jira.eso.org/browse/PIPE-5354)
        (https://jira.eso.org/browse/PIPE-5407)
        (https://jira.eso.org/browse/PIPE-5415)

== v1.3.4 ==
''(25.07.2014, by ESO)''
 * Recipes
    - Replaced kmo_dark, kmo_flat, kmo_wave_cal, kmo_illumination by
        kmos_dark, kmos_flat, kmos_wave_cal, kmos_illumination
    - kmos_flat made faster - The OFF frames are combined once at the beginning,
        not for each angles again and again. Additional optimisations applied.
        kmos_flat takes now 5 min instead of 13 min to process a standard FLAT
        data set.
        (https://jira.eso.org/browse/PIPE-4916)
    - kmo_illumination could not find kmo_calc_edgepars() (was renamed to
        kmos_calc_edgepars() and made static).
        Tests did not catch it because it only occurs when using
        FLAT_EDGE calibration frames. Tests updated.
        (https://jira.eso.org/browse/PIPE-5281)
        (https://jira.eso.org/browse/PIPE-5292)
    - kmo_illumination products header keywords did inherit from the
        input FLAT_EDGE keywords instead of inheriting from the input
        RAW files.
        (https://jira.eso.org/browse/PIPE-5340)
    - kmos_dark 15% faster than kmo_dark
      kmos_illumination 10% faster than kmo_illumination
        (https://jira.eso.org/browse/PIPE-5026)
    - kmo_wave_cal crashes when the last K band line cannot be
        identified. This happened from time to time.
        REF_LINES calibration file has been updated with a new Reference
        line for K band on detector 2. The rightmost reference lines (the
        one missed in the reported failures) is replaced by a more central one.
        kmos_gen_reflines utility has been added to help producing the
        REFLINES calibration file in the future.
        (https://jira.eso.org/browse/PIPE-5273)
        (https://jira.eso.org/browse/PIPE-5209)
        (https://jira.eso.org/browse/PIPE-5169)
        (https://jira.eso.org/browse/PIPE-5038)

 * Reflex
    - Update the Workflow to integrate new recipes names and options

 * Tests
    - All compiler warnings have been removed. Also the unit tests
        messages were removed.
        (https://jira.eso.org/browse/PIPE-5036)
    - Unit test kmclipm_test_rtd is disabled for the moment as it is too
        long and requires too much memory
        (https://jira.eso.org/browse/PIPE-5121)

== v1.3.3 ==
''(16.06.2014, by ESO)''
 * Recipes
    - Fixed the blocking issue that prevents kmo_flat to process Detector 2 since IFU 15 was disabled.

== v1.3.2 ==
''(07.05.2014, by ESO)''
 * Recipes
    - Incorrect angles reported by kmo_wave_cal when checking XCAL YCAL compatibility
        (https://jira.eso.org/browse/PIPE-5035)
    - Recipes with xcal interpolation give misleading messages
        (https://jira.eso.org/browse/PIPE-5065)
    - Failed telluric correction in KMOS recipe kmo_sci_red
        (https://jira.eso.org/browse/PIPE-5092)
    - kmo_sci_red fails with -sky_tweak and -obj_sky_table used together
        (https://jira.eso.org/browse/PIPE-5165)

 * Static Calibration Files
    - CRPIX1 values have wrong type in OH lines and solar spectra
        (https://jira.eso.org/browse/PIPE-5159)
        (https://jira.eso.org/browse/PIPE-5158)

 * DFS related
    - easySPARK scripts explicitely call option --suppress-prefix=TRUE
        (https://jira.eso.org/browse/PIPE-5136)

 * Unit/Regression Tests
    - NRI regression tests are fixed.
        (https://jira.eso.org/browse/PIPE-4909)

 * Reflex
    - Missing relation in "Collapse Datacube" Actor
        (https://jira.eso.org/browse/PIPE-5047)
    - kmo_wave_cal interactive actor fails with python 2.6 (string formatting)
        (https://jira.eso.org/browse/PIPE-5164)
    - Provenance Explorer needs to be controled (disabled/enabled)
        (https://jira.eso.org/browse/PIPE-5147)
    - Empty IFUs are identified (in Black) in the IFU selector
        (https://jira.eso.org/browse/PIPE-5039)
    - Add a Legend to the spectrum plots when they have multiple plots
        (https://jira.eso.org/browse/PIPE-5039)
    - The FITS viewer of the provenance explorer is always fv. This
    problem is reported to the Reflex team under:
        (https://jira.eso.org/browse/DFS-9849)
    - Change the imethod default value to MS in the kmo_std_star
        interactive actor in order to have the Noise produced. Whenever
        the noise is available, overplot it on the star spectrum.
        (https://jira.eso.org/browse/PIPE-5039)
    - Image scaling dynamics changed to improve star image
        (https://jira.eso.org/browse/PIPE-5039)

== v1.3.1 ==
''(xx.xx.2014, by ESO)''
???

SPARK Change Log

v1.3.6

(07.11.2014, by ESO)

  • Recipes
    • - kmos_illumination is failing on a single data set on detector 3.
      • The reason is an unfortunate slitlets sanity check that fails on this particular data. The check is still considered good and the pipeline remains unchanged

        (https://jira.eso.org/browse/PIPE-5387)

      - kmo_sky_tweak is failing on a single data set with endless - Extracted spectra contain NaN values when extracting with a mask
  • Calibration Scripts
    • - easySPARK_calibration fails if the OB crashed, although all the
      • data ave available This ticket is adressing the easySPARK scripts issues in general. A deep analysis of the needs and the reported issues (4 cases of emergency support for Paranal Science operations occured in 2014) lead to a full re-write. The new algorithm for files automatic selection / grouping and recipes execution has been discussed and agreed upon with Paranal Science Operations during the October 2014 visit. The easyPLUG (graphical tool - 4000 l. of perl) and the easySPARK scripts (3000 l. of shell) software are not delivered any more from this version on. The calibration files generation needed in Paranal is exclusively covered by the new script kmos_calib.py (500 l. of python)

v1.3.5

(08.09.2014, by ESO)

v1.3.4

(25.07.2014, by ESO)

v1.3.3

(16.06.2014, by ESO)

  • Recipes
    • - Fixed the blocking issue that prevents kmo_flat to process Detector 2 since IFU 15 was disabled.

v1.3.2

(07.05.2014, by ESO)

v1.3.1

(xx.xx.2014, by ESO) ???

v1.3.0

(07.03.2014, 1st release by ESO)

  • kmo_sci_red, kmo_sky_tweak

    • Fixed bug: Background fitting was weak in some bands (specially in YJ). Background fitting has been improved and stabilized.
    • Fixed bug: In IZ-band occasional crashes were reported.
  • kmo_sci_red

    • Fixed bug: Providing just one SCIENCE frame and an obj_sky_table indicating to subtract skies from another IFU should work now (before sky hasn't been subtracted without notification).
  • kmo_sky_tweak

    • Fixed bug: frames with inactive IFUs are processed correctly now.
  • kmo_std_star

    • Fixed bug: the areas of NaN values in the STAR_SPEC and TELLURIC products have been fixed.
  • kmo_make_image

    • Fixed bug: Take into account range-parameter even if no OH-line spectrum is provided (has just been ignored before).
    • Fixed bug: Fixing undescriptive error message when CRVAL1 keyword in OH-line spectrum was an integer instead of a double value.

v1.2.8

(12.12.2013)

  • kmo_sci_red

    • Adding the option to apply a fixed velocity offset when reconstructing cubes.
    • Adding parameter --save_interims which can be used only together with --no_sky_tweak. Interim reconstructed object and sky cubes are saved to disk as well.
    • Adding the possibility to subtract skies from objects from different IFUs (using the generated obj_sky_table.txt)
  • kmo_std_star

    • Bug fix: reducing templates with the 3-star configuration produced a segmentation fault
  • kmo_wave_cal

    • Bug fix: in rare cases an error occurred calculating the QC parameters (singular matrix)

v1.2.7

(25.11.2013)

  • kmo_combine, kmo_sci_red

    • New additional output frame EXP_MASK: For every combined output frame there is now as well an exposure time mask which indicates for every spatial pixel how many input frames were taken into account for combining.
  • kmo_sci_red, kmo_multi_reconstruct

    • The object-sky-association table is stored to disk as an intermediate text file (obj_sky_table.txt). It can be edited to alter the associated skies and then read into the recipe again to apply it (esorex kmo_sci_red --obj_sky_table=obj_sky_table.txt <...>).

  • kmo_std_star

    • The signal level of the extracted spectrum (in the masked area) is scaled to the same level as the spectrum of the whole IFU (which is used for zeropoint and throughput calculations). The user can now do his own calculations on zeropoint, extracted spectrum etc., since they match now.
    • New product: NOISE_SPEC contains in any case the shot noise [sqrt(counts*gain)/gain]. If the exposures have been taken with template KMOS_spec_cal_stdstarscipatt, then an additional noise component is added in: All existing sky exposures for an IFU are subtracted pairwise, spectra are extracted and the std deviation is calculated.
  • kmo_sci_red

    • Bug fix: "checkerboard pattern" effect: rarely it could happen, that single spaxels at the border of an IFU were spectral misfitted (only with --imethod=CS). This effect normally to be overlooked, has a strong effect when these cobes are combined after reconstruction. Then some kind of checkerboard pattern could emerge in the corners of arbitrary IFUs.
  • kmo_combine

    • Bug fix: when --ifus parameter wasn't defined and if the first IFU wasn't used the recipe would create a segmentation fault
    • Bug fix: when working with the --ifus parameter, the EXTNAME keyword could get too long and the extension descriptor DATA/NOISE could be cut off. The keyword is now truncated and DATA or NOISE is appended in every case. The same naming convention applies to the craeted filenames (except that DATA/NOISE isn't added).

v1.2.6

(30.10.2013, new kit created on 12.11.2013 du to installation problems)

  • kmo_illumination_flat

    • New recipe creating an illumination correction based on FLAT_ON frames.
  • kmo_reconstruct

    • Bug fix: wrong calibration angles have been displayed, but the right ones were processed (probably introduced with v1.2.5)

v1.2.5

(14.10.2013)

  • kmo_flat

    • Bug fix: Enhanced edge detection taking into account deactivated IFUs which are illuminated quite strongly.
  • kmo_multi_reconstruct

    • Accepts now like kmo_sci_red MASTER_FLAT and OH_SPEC frames, does proper sky subtraction and optionally flux/background calibration. Telluric correction is possible if the objects are on the same IFU for all exposures. Maps can't be reduced.
  • kmo_sci_red

    • Bug fix: Object names containing [ or ] couldn't be processed.
  • sky_tweak

    • Bug fix: With continuum missing on one side of the spectrum the code failed.
  • easySPARK_calibration.sh

    • The user is not asked anymore if the data should be processed in parallel/serial. This option can be set via a parameter.
    • After completion of processing a set of tests is executed to ensure integrity of calibration products (easySPARK_test.sh)
    • Added version numbers to easySPARK-scripts

v1.2.4

(bug-fix release, 07.08.2013)

  • kmo_wave_cal

    • Bug fix: Sometimes DET_IMG_WAVE product wasn't calculated.
  • SPARK-kit installation

    • Bug fix: On MacOSX compilation of the kit failed.

v1.2.3

(bug-fix release, 02.08.2013)

  • kmo_sky_tweak

    • New, advanced recipe for OH-line subtraction
  • kmo_sci_red

    • Bug fix: When using reflex it happend that only the IFUs containing the std-stars were processed (in v1.2.2 this can be fixed by omitting STD_IMAGE from sof)
    • Bug fix: TELLURIC frames generated with data from the 3star-template couldn't be processed if the std-stars weren't at the default IFUs
    • Enhanced handling of telluric corrections in general
  • kmo_flat

    • Fix: Due to the corrupted IFU 16, it occured in some FLAT_ON frames that IFU 15 was vignetted strongly and not all edges could be identified. This is detected now and IFU 15 will be omitted.
  • kmo_illumination

    • Bug fix: When data contains IFUs marked as NOTUSED and FLAT_EDGE frame is provided, the recipe just fails without any notice. Without FLAT_EDGE frame the recipe runs through
  • kmo_fits_strip

    • Bug fix: Parameters noise and empty failed on sci_combine-products
    • Added extension-parameter: Provide absolute extension numbers to strip all other extensions
  • kmo_fits_check

    • Now prints as well the number of empty extensions
  • kmo_reconstruct, kmo_sci_red, kmo_std_star, kmo_illumination

    • Increased verbosity on picked calibration frames. Emitting warnings when angles differ too much
    • xcal_interpolation is switched off automatically when interval of neighbouring angles is larger than 65deg (for 6 equidistant rotator angles it would be 60deg)
  • easySPARK-scripts

    • Fixed bug: Now the frames needn't anymore be organised in directories with names like YYYY-MM-DD
    • Fixed bug: Now the scripts work as well on MacOSX

v1.2.2

(intermediate release for Software Verification 1, 29.06.2013)

  • reconstruction

    • Bug fix: the implementation of xcal_interpolation accidentially marked internally some areas in the calibration frames as bad. Since these areas had a width of 1 pixel, they were just interpolated from the neighbourhood. Now the values are a little bit more correct.
    • Bug fix: the implementation of cubic spline (CS) interpolation in the reconstruction of data cubes led from time to time to misfitted spectra, mostly at the borders of the IFUs. Pronounced spectral bell-curves were the result.
  • kmo_std_star

    • Bug fix: Due to the NGC using a wrong data type for CRPIX, CRVAL and CD keywords (integer instead of double) in the RAW frames, no standard stars could be processed.

v1.2.1

(bug fix, 20.06.2013)

  • kmo_multi_reconstruct

    • The new calibration format containing multiple angles was ignored processed wrongly. For the calibration frames always the first angle present has been used in the past.

v1.2.0

(Release for Software Verification, 19.06.2013)

  • Reference Manual

    • Updated version.
  • several bug fixes related to SW release management

v1.1.5

(intermediate release, 05.06.2013)

  • kmo_sci_red

    • Bug fix: Failed when OH_SPEC was provided
  • reconstruction

    • No more spikes at the beginning and end of reconstructed spectra (with cubic spline)

v1.1.4

(intermediate release, 23.05.2013)

  • kmo_reconstruct, kmo_illumination, kmo_std_star, kmo_sci_red, kmo_shift, kmo_rotate, kmo_combine

    • Enhancements on flux-conservation. It is switched off as default except in kmo_std_star where we can expect to have a bright object in the IFU.
  • kmo_reconstruct

    • Fixed bug in OH-line matching.
    • OH-line matching based on a reference OH-spectrum improving object-sky subtraction (like in kmo_reconstruct)
  • kmo_sci_red

    • Enhancements on background-removal for mapping-mode. This feature is off per default since it works best for maps with sparse objects in it.
    • OH-line matching based on a reference OH-spectrum improving object-sky subtraction (like in kmo_reconstruct)
  • kmo_reconstruct, kmo_std_star, kmo_sci_red

    • Implemented interpolation of xcal-frames for angles in between the frames. This feature is used as default. To switch it off, set the parameter --xcal_interpolation=FALSE
  • kmo_make_image

    • Fixed bug: couldn't collapse a SCI_COMBINED frame.
  • RTD

    • Object- and sky-frame are now first subtracted and then reconstructed (Speed improvement).
  • kmo_extract_spec, kmo_make_image, kmo_stats

    • Fixed bug regarding gasgano: any category for the 1st input frame is accepted now.

v1.1.3

(intermediate release, 13.05,2013)

  • kmo_illumination

    • added --pix_scale parameter for arbitrary pixel resolution in reconstructed cubes
  • kmo_reconstruct, kmo_sci_red

    • Flux of reconstructed cubes is scaled according to pixel_scale
  • SPARKplug

    • Fixed bug: Copying selected frames to lower panel should work now as well on MacOS
  • kmo_sci_red

    • Fixed bug with --no_subtract: Accepts now as well sky-only frames
  • easySPARK-scripts

    • Fixed bugs with ESO archive filename scheme

v1.1.2

(intermediate release, 08.05.2013)

  • kmo_combine

    • Changed default behaviour: if no IFU or name is specified, all target names are combined and saved (no mapping-mode as default anymore)
  • kmo_reconstruct

    • OH-line matching based on a reference OH-spectrum improving object-sky subtraction
    • added --pix_scale parameter for arbitrary pixel resolution in reconstructed cubes
  • kmo_sci_red

    • added --no_combine option
    • added --no_subtract option (implies --no_combine because otherwise IFUs containing skies would be combined with ones with objects in it. With kmo_combine the products can be combined individually afterwards)
    • added --pix_scale parameter for arbitrary pixel resolution in reconstructed cubes
    • if just a single SCIENCE frame is provided, ALL IFUs (of type R, O and S) are processed

v1.1.1

(intermediate release, 23.04.2013)

  • bug fix:

    • All recipes doing reconstruction were failing when reading the calibration frames if angle=0 deg wasn't present
  • kmo_std_star

    • Doesn't crash anymore when star type is defined wrongly in header keywords
  • kmo_std_star, kmo_sci_red

    • Processes now as well R (along with O and S) in header keyword ESO OCS ARMx TYPE
  • kmo_combine, kmo_sci_red

    • Added parameter --edge_nan. Sets top/bottom row to NaN, improves results when dithering (like with SINFONI data).
  • kmo_reconstruct, kmo_sci_red

    • First implementation of improved flux conservation (still to be improved).
  • kmo_sci_red

    • Added parameter --background. Activate for background removal (to be improved).

v1.1.0

(Release for COMM3, 19.03.2013)

  • kmo_illumination

    • Fixed edge effects in ILLUM_CORR. Takes now FLAT_EDGE frame as input to correlate MASTER_FLAT and FLAT_SKY frames. Additional output SKYFLAT_EDGE.
  • kmo_reconstruct, kmo_multi_reconstruct, kmo_illumination, kmo_sci_red

    • Emit warning if XCAL, YCAL and LCAL frames don't originate from the same calibration set.
  • kmo_sci_red

    • Emit warning if the parameters b_samples, b_start and b_end aren't the same for LCAL and TELLURIC frames.

v1.0.9

(intermediate release for USM, not published here, 13.03.2013)

  • kmo_illumination

    • Ignores now the first exposure of the KMOS_spec_cal_skyflat template which is just for acquisition. With the parameter --add-all it can be forced to be included.
    • Added band-specific spectral ranges used to collapse the cubes in order to create the skies.
  • kmo_std_star

    • accepts lower case star types
    • corrected WCS in headers of products
    • QC THRUPUT and QC ZPOINT are now calculated on the whole area of the IFU and not only on the masked area
    • QC STD TRACE are now calculated based on the difference of two specified spectral ranges. Before, the whole spectrum was taken into account resulting in inadequate results.
    • Fixed WCS in product headers as well as EXTNAME keyword which caused errors when processing the products in the next steps
  • easySPARK

    • Fixed bugs in easySPARK_illumination.sh and easySPARK_std_star.sh
    • Added easySPARK_sci_red.sh
  • kmo_sci_red

    • Increased verbosity on displayed shift parameters
    • For mapping-mode added --fast-mode parameter for a quick preview based on collapsed cubes
    • Added telluric tweaking for 3stdstar-template and if 24-template isn't complete
  • kmo_combine

    • Increased verbosity on displayed shift parameters
  • kmo_fits_strip

    • When used with --angle parameter and with calibration files the NAANGLE keywords are now handled correctly
    • Added parameter --empty to remove empty extensions

v1.0.8

(Intermediate release for software developers at USM, 20.02.2013)

  • kmo_wave_cal'

    • no BADPIXEL_FLAT frame needed anymore
    • fixed memory leaks
  • Added "file_extension" parameter

    • If set, the filenames of the products can be altered. The behaviour for the recipes differs:
      • kmo_arithmetic: provide an arbitrary string to append
      • kmo_dark: set to TRUE or FALSE to append EXPTIME keyword

      • kmo_reconstruct: set to TRUE or FALSE to append OBS_ID keyword

  • kmo_flat

    • optimised memory usage, needs now less RAM, a bit slower due to disk I/O

v1.0.7

(08.01.2013)

  • organisation of calibration files

    • XCAL/YCAL/LCAL contain now all rotator angles in a single file (e.g. xcal_HHH_HHH

KMOS-SPARKwiki: Change Log (last edited 2014-11-11 08:05:24 by obelix)