Differences between revisions 2 and 13 (spanning 11 versions)
Revision 2 as of 2021-02-16 08:19:47
Size: 5641
Comment:
Revision 13 as of 2021-02-16 12:01:53
Size: 10561
Comment:
Deletions are marked like this. Additions are marked like this.
Line 4: Line 4:
'''Start of meeting: 2021-02-16 12:30 <- meeting starts earlier this time and will end before 14:00''' '''Start of meeting: 2021-02-16 12:30 <<<< meeting starts earlier this time and will end before 14:00'''
Line 9: Line 9:

----

'''''Previous meeting:'''''
Line 19: Line 15:

   * Preprocessing / NRTA

   * eSASS tasks (ongoing/planned) / current processing errors - [[https://wiki.mpe.mpg.de/eRosita/EroCat/eSASSdevel|eSASSdevel page]] / [[https://wiki.mpe.mpg.de/eRosita/pipelineErrorTracking|Error tracking page]]
     * eRASS1/947 code updates/code freeze
       * perhaps small evprep updates
       * expmap:
         * parallel version (7 cores)
         * withinputmaps=yes
         * testing?
       * catprep: making number of bands variable (1..5) - in progress
       * detection task updates (ongoing, perhaps ready for 947)
         * positional error / count error 0
         * upper/lower count rate errors - will also be propagated by catprep
         * -> ermldet new task version
       * ftfindhotpix / badpix / caldb access in general
         * accessing highest validity date file vs all overlapping file
         * TMIN/TMAX columns not use in current version
         * Ian will make ero_cifsl return files in decending validity start date order
       * Temperature dependent energy calibration
         * Konrad will provide update of task energy (will require new caldb file)
         * Pattern update: read badpix start and end times
     * srctool vs. apetool exposure (help desk question)
     * fftw version (Alain)
       * different ermldet results on old/new machines may be related to fftw version (results with v3.3.9 are ok)
         * v3.3.8 in current user release should be upgraded
     * Pipeline tasks (shadow developer/pipeline operator)
       * eropipe, proc, prexec - Alain
       * telstage, dalist, rmvers - Miriam
       * test environment, current task versions -> eSASSdevel/task archive
     * Recent task updates:
       * [[https://lists.mpe.mpg.de/cgi-bin/mailman/private/erosita-sass/2021-February/001151.html|erolib-2.40, evtool-2.19, evprep-3.19]]

   * Pipeline / ongoing processing
     * [[https://wiki.mpe.mpg.de/eRosita/PipeProc|processing wiki page]]
     * performance issues?
       * replacement for get_dir routine?
     * 947 test runs
       * eRASS1/2, eRASS:2 (first 100 eDays) are in archive
         * HiPS (eRASS1/2), DAtool, ProcStat (ok)
           * DAtool (947): ImageTot -> Image, energy bands: 0-9, a-d for e?01..., e?02..., s?02...
           * HiPS energy bands?
         * Astrometry?
     * eRASS2/3 946 exposure deficit
       * use latest HK file versions + tel-1.24 (telgti border issues)
       * fixed in 947 / no full 946 rerun of affected data planned
     * Missing soft sources (Frank)
       * pre-946 processing versions contained CalPV test scans
       * confirmation: rerun test scans in 946 pipeline (or 001?)
     * TM7 bad pixels
       * caldb was updated, eDays 46199+ restarted
     * 949 TEL chain
       * for Konrad's temperature dependent energy calibraton
       * run TEL chain up until pattern
       * CalPV + recent data

   * Data access
     * eRASS event files / images
       * FWC data

  * EDR
    * Web pages, documentation
    * Validation of CalPV/c001 data: guidlines by eSASS team

  * DAtool / ProcSTat
    * access restrictions for data request logs + project descriptions (discussion ongoing - details tbd)
    * providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
    * move simple copying request to Python / should everything be moved to Python? - ongoing
    * DAtool / eROSE : requires change in frontend

  * Longterm activities
    * '''[?]''' Moving task archive to git
    * evprep
      * '''[9]''' evprep: space invaders event flag
        * evtool should update the gti according to this flag
        * post-processing: frames with many counts?
      * BTI/GTI:
        * PDF for CalPV linked on wiki page (done?)
        * '''[7]''' one PDF per eRASS in wiki
    * ermldet
      * '''[8]''' ca. 2% of sources have positional error 0, some also have count error 0
        * can't be fixed quickly, may need replacement of algorithm
        * Johan's proposal? Georg to discuss with Johan (long timescale project)
    * Time dependent RMFs/ARFs (srctool?)
      * '''[8]''' RMF
      * '''[7]''' ARF
    * '''[6]''' Background analysis: projecting closed data onto the sky (XMM SAS)
    * '''[4]''' ersensmap: implement pointing PSF
    * '''[3]''' ftfindhotpix: test and update algorithm
    * '''[1]''' Light leak in eSASS/pipeline?
    * '''[1]''' DAtool: filtering catalogs (separate from DAtool)?

----

'''''Previous meeting:'''''

   * Open actions from previous developers meetings
     * Act20200929/1_all make sure html task descriptions are current and consistent with user release
       * updates by Jeremy and Ian (evtool, expmap, catprep)
       * still missing: Hermann (radec2xy), Georg (detection + evatt)
     * Act20201006/1_SF clean event file description, u pdate source list description

Weekly eSASS developers meeting

Start of meeting: 2021-02-16 12:30 <<<< meeting starts earlier this time and will end before 14:00

Zoom link: https://zoom.us/j/97439409844?pwd=YXVkK0hBMnpybm5tL2VyYTh2Qld2UT09

Participants: Ian Stewart, Hermann Brunner, Miriam Ramos, Alain Gueguen, Konrad Dennerl, Mara Salvato, Sabine Osterhage, Susanne Friedrich, Jeremy Sanders. Chandeyee Maitra, Andrea Merloni, Michael Freyberg

  • Open actions from previous developers meetings
    • Act20200929/1_all make sure html task descriptions are current and consistent with user release
      • updates by Jeremy and Ian (evtool, expmap, catprep)
      • still missing: Hermann (radec2xy), Georg (detection + evatt)
    • Act20201006/1_SF clean event file description, update source list description
  • Preprocessing / NRTA
  • eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page

    • eRASS1/947 code updates/code freeze
      • perhaps small evprep updates
      • expmap:
        • parallel version (7 cores)
        • withinputmaps=yes
        • testing?
      • catprep: making number of bands variable (1..5) - in progress
      • detection task updates (ongoing, perhaps ready for 947)
        • positional error / count error 0
        • upper/lower count rate errors - will also be propagated by catprep
        • -> ermldet new task version

      • ftfindhotpix / badpix / caldb access in general
        • accessing highest validity date file vs all overlapping file
        • TMIN/TMAX columns not use in current version
        • Ian will make ero_cifsl return files in decending validity start date order
      • Temperature dependent energy calibration
        • Konrad will provide update of task energy (will require new caldb file)
        • Pattern update: read badpix start and end times
    • srctool vs. apetool exposure (help desk question)
    • fftw version (Alain)
      • different ermldet results on old/new machines may be related to fftw version (results with v3.3.9 are ok)
        • v3.3.8 in current user release should be upgraded
    • Pipeline tasks (shadow developer/pipeline operator)
      • eropipe, proc, prexec - Alain
      • telstage, dalist, rmvers - Miriam
      • test environment, current task versions -> eSASSdevel/task archive

    • Recent task updates:
  • Pipeline / ongoing processing
    • processing wiki page

    • performance issues?
      • replacement for get_dir routine?
    • 947 test runs
      • eRASS1/2, eRASS:2 (first 100 eDays) are in archive
        • HiPS (eRASS1/2), DAtool, ProcStat (ok)

          • DAtool (947): ImageTot -> Image, energy bands: 0-9, a-d for e?01..., e?02..., s?02...

          • HiPS energy bands?
        • Astrometry?
    • eRASS2/3 946 exposure deficit
      • use latest HK file versions + tel-1.24 (telgti border issues)
      • fixed in 947 / no full 946 rerun of affected data planned
    • Missing soft sources (Frank)
      • pre-946 processing versions contained CalPV test scans
      • confirmation: rerun test scans in 946 pipeline (or 001?)
    • TM7 bad pixels
      • caldb was updated, eDays 46199+ restarted
    • 949 TEL chain
      • for Konrad's temperature dependent energy calibraton
      • run TEL chain up until pattern
      • CalPV + recent data
  • Data access
    • eRASS event files / images
      • FWC data
  • EDR
    • Web pages, documentation
    • Validation of CalPV/c001 data: guidlines by eSASS team
  • DAtool / ProcSTat
    • access restrictions for data request logs + project descriptions (discussion ongoing - details tbd)
    • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
    • move simple copying request to Python / should everything be moved to Python? - ongoing
    • DAtool / eROSE : requires change in frontend
  • Longterm activities
    • [?] Moving task archive to git

    • evprep
      • [9] evprep: space invaders event flag

        • evtool should update the gti according to this flag
        • post-processing: frames with many counts?
      • BTI/GTI:
        • PDF for CalPV linked on wiki page (done?)
        • [7] one PDF per eRASS in wiki

    • ermldet
      • [8] ca. 2% of sources have positional error 0, some also have count error 0

        • can't be fixed quickly, may need replacement of algorithm
        • Johan's proposal? Georg to discuss with Johan (long timescale project)
    • Time dependent RMFs/ARFs (srctool?)
      • [8] RMF

      • [7] ARF

    • [6] Background analysis: projecting closed data onto the sky (XMM SAS)

    • [4] ersensmap: implement pointing PSF

    • [3] ftfindhotpix: test and update algorithm

    • [1] Light leak in eSASS/pipeline?

    • [1] DAtool: filtering catalogs (separate from DAtool)?


Previous meeting:

  • Open actions from previous developers meetings
    • Act20200929/1_all make sure html task descriptions are current and consistent with user release
      • updates by Jeremy and Ian (evtool, expmap, catprep)
      • still missing: Hermann (radec2xy), Georg (detection + evatt)
    • Act20201006/1_SF clean event file description, u pdate source list description
  • Preprocessing / NRTA
    • Noise/offset maps?
  • eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page

    • eRASS1/947 code updates/code freeze (Ian's email)
      • perhaps small evprep updates
    • expmap:
      • parallel version (7 cores)
      • second order parallelization (by GTI)
      • withinputmaps=yes
      • testing?
    • catprep: making number of bands variable (1..5) - in progress
    • detection task updates (ongoing, perhaps ready for 947)
      • clobber option
      • improved RADEC_ERR, upper/lower count rate errors
    • Pipeline tasks (shadow developer/pipeline operator)
      • eropipe, proc, prexec - Alain
      • telstage, dalist, rmvers - Miriam
      • test environment, current task versions -> eSASSdevel/task archive

    • Recent task updates:
      • expmap: v2.1.1-3, v2.2, v1.33.4
  • Pipeline / ongoing processing
    • processing wiki page

    • performance issues?
      • replacement for get_dir routine?
      • ds53 speed issues?
    • 947 test runs
      • eRASS1/2, eRASS:2 (first 100 eDays) are in archive
        • HiPS (eRASS1/2), DAtool, ProcStat (ok)

          • DAtool (947): ImageTot -> Image, energy bands: 0-9, a-d for e?01..., e?02..., s?02...

          • HiPS energy bands?
        • Astrometry?
    • eRASS3 (946) exposure deficit
      • caused (in part) by incomplete/not-current HK file versions - fixed
      • telgti issues at border of planning files (fixed in tel-1.24)
      • 46008 and 46085 rerun in eRASS3 946 processing - HiPS maps?
      • similar exposure deficits in eRASS1/2
    • Missing soft sources (Frank)
      • Ian, Chandreyee, Frank investigated
        • probable cause: previous processings contained CalPV test scans
        • confirmation: rerun test scans in 946 pipeline (or 001?)
    • bad pixels
      • TM7:
        • 42 bad pixels in caldb (onboard vs. caldb?)
          • onboard: 40 bad pixels
        • How to proceed with eDay 46199?
          • Susanne will set vsd to beginning of eROday 46199
    • Temperature dependence in task energy (HK access)
      • should go in 947 pipeline
      • Konrad proved list of eROdays to Hermann - TEL_949 chains currently running
    • Zip pattern analysis
      • Susnanne + Chandreyee will discuss
    • Discrepancies in source list of demo script runs / ermldet uninitialized variables
      • Still different results of same binaries on different machines
      • Different from pervious problem, all sources are affected (parameter are sightly different) - high priority
      • Confirm whether ermldet patch changes the eFEDS source list (Teng?)
  • DAtool
    • access restrictions sto data request logs + project descriptions (discussen ongoing- details tbd)
    • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
    • move simple copying request to Python / should everything be moved to Python? - ongoing
    • DAtool / eROSE interface: requires change in frontend
  • Longterm activities
    • [?] Moving task archive to git

    • evprep
      • [9] evprep: space invaders event flag

        • which flag location: bit 3, parameter in pipeline needs to be turned on
        • evtool should update the gti according to this flag
      • post-processing: frames with many counts?
    • BTI/GTI:
      • PDF for CalPV linked on wiki page (done?)
      • [7] one PDF per eRASS in wiki

    • ermldet
      • [8] ca. 2% of sources have positional error 0, some also have count error 0

        • can't be fixed quickly, may need replacement of algorithm
      • Johan's proposal? Georg to discuss with Johan (long timescale project)
    • Time dependent RMFs/ARFs (srctool?)
      • [8] RMF

      • [7] ARF

    • [6] Background analysis: projecting closed data onto the sky (XMM SAS) -> 6

    • [4] ersensmap: implement pointing PSF

    • ftfindhotpix: test and update algorithm
    • [1] Light leak in eSASS/pipeline?

    • [1] DAtool: filtering catalogs (separate from DAtool)? -> 1

EROSITAwiki: eSASSdevel_210216 (last edited 2021-03-02 08:47:42 by HermannBrunner)