Differences between revisions 7 and 8
⇤ ← Revision 7 as of 2021-02-09 11:05:22 →
Size: 8755
Editor: HermannBrunner
Comment:
← Revision 8 as of 2021-02-09 11:26:22 → ⇥
Size: 10515
Editor: HermannBrunner
Comment:
Deletions are marked like this. Additions are marked like this.
Line 38: Line 38:
       * replacement for get_dir routine?
       * ds53 speed issues?
Line 58: Line 60:
       * Konrad proved list of eROdays to Hermann - HB will run eROdays in 947 pipeline!!        * Konrad proved list of eROdays to Hermann - TEL_949 chains currently running
Line 64: Line 66:
       * Confirm whether ermldet patch changes the eFEDS source list (Teng?)
    * pipeline (eropipe2) machine utilization
       * potential improvements
    * MIP threshold increase (get threshold from HK)
       * evprep-3.18/3.11.3		        * Confirm whether ermldet patch changes the eFEDS source list (Teng?)
Line 71: Line 69:
    * HiPS / ds53 issues / processing speed
       * migrating to He4ero?
    * EDR
      * Web pages, documentation
      * Validation of CalPV/c001 data: guidlines by eSASS team
        * https://wiki.mpe.mpg.de/eRosita/guide_validationEDR

   * DAtool
     * access restrictions sto data request logs + project descriptions
     * providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
     * move simple copying request to Python / should everything be moved to Python? - ongoing
     * DAtool / eROSE interface: requires change in frontend


 * Longterm activities
     * '''[?]''' Moving task archive to git
     * evprep
       * '''[9]''' evprep: space invaders event flag
         * which flag location: bit 3, parameter in pipeline needs to be turned on
         * evtool should update the gti according to this flag
       * post-processing: frames with many counts?
     * BTI/GTI:
       * PDF for CalPV linked on wiki page (done?)
       * '''[7]''' one PDF per eRASS in wiki
     * ermldet
       * '''[8]''' ca. 2% of sources have positional error 0, some also have count error 0
         * can't be fixed quickly, may need replacement of algorithm
       * Johan's proposal? Georg to discuss with Johan (long timescale project)
     * Time dependent RMFs/ARFs (srctool?)
       * '''[8]''' RMF
       * '''[7]''' ARF
     * '''[6]''' Background analysis: projecting closed data onto the sky (XMM SAS) -> 6
     * '''[4]''' ersensmap: implement pointing PSF
     * ftfindhotpix: test and update algorithm
     * '''[1]''' Light leak in eSASS/pipeline?
     * '''[1]''' DAtool: filtering catalogs (separate from DAtool)? -> 1

  * DAtool
     * providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
     * move simple copying request to Python / should everything be moved to Python? - ongoing
     * DAtool / eROSE interface: requires change in frontend

Weekly eSASS developers meeting

Start of meeting: 2021-02-09 13:30 (1:30pm)

Zoom link: https://zoom.us/j/97439409844?pwd=YXVkK0hBMnpybm5tL2VyYTh2Qld2UT09

  • Open actions from previous developers meetings
    • Act20200929/1_all make sure html task descriptions are current and consistent with user release
      • updates by Jeremy and Ian (evtool, expmap, catprep)
      • still missing: Hermann (radec2xy), Georg (detection + evatt)
    • Act20201006/1_SF clean event file description, update source list description
  • Preprocessing / NRTA
    • Noise/offset maps?

  • eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page

    • eRASS1/947 code updates/code freeze (Ian's email)
    • expmap:
      • parallel version (7 cores)
      • second order parallelization
      • withinputmaps=yes
      • testing?
    • catprep: making number of bands variable (1..5) - in progress
    • detection task updates (ongoing, perhaps ready for 947)
      • clobber option
      • improved RADEC_ERR, upper/lower count rate errors
    • Pipeline tasks (shadow developer/pipeline operator)
      • eropipe, proc, prexec - Alain
      • telstage, dalist, rmvers - Miriam

      • test environment, currnent task versions -> eSASSdevel/task archive

    • Recent task updates:
      • expmap: v2.1.1-3, v2.2, v1.33.4
  • Pipeline / ongoing processing

    • processing wiki page

    • performance issues?
      • replacement for get_dir routine?
      • ds53 speed issues?
    • 947 test runs
      • eRASS1/2, eRASS:2 (first 100 eDays) are in archive

        • HiPS (eRASS1/2), ProcStat (ok), DAtool

          • DAtool (947): ImageTot -> Image, energy bands: 0-9, a-d for e?01..., e?02..., S?02...

        • Astrometry?
    • eRASS3 (946) exposure deficit
      • caused (in part) by incomplete/not-current HK file versions - fixed
      • telgti issues at border of planning files (fixed in tel-1.24)
      • 46008 and 46085 rerun in eRASS3 946 processing - HiPS maps?
      • similar exposure deficits in eRASS1/2
    • Missing soft sources (Frank)
      • Ian, Chandreyee, Frank investigated
        • probable cause: previous processings contained CalPV test scans
        • confirmation: rerun test scans in 946 pipeline (or 001?)
    • bad pixels
      • TM7:
        • 42 bad pixels in caldb (onboard vs. caldb?)
        • How to proceed with eDay 46199?
    • Temperature dependence in task energy (HK access)
      • should go in 947 pipeline
      • Konrad proved list of eROdays to Hermann - TEL_949 chains currently running
    • Zip pattern analysis
      • updating badpixel / bti in caldb
    • Discrepancies in source list of demo script runs / ermldet uninitialized variables
      • Still different results of same binaries on different machines
      • Different from pervious problem, all sources are affected (parameter are sightly different) - high priority
      • Confirm whether ermldet patch changes the eFEDS source list (Teng?)
  • DAtool
    • access restrictions sto data request logs + project descriptions
    • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
    • move simple copying request to Python / should everything be moved to Python? - ongoing
    • DAtool / eROSE interface: requires change in frontend
  • Longterm activities

    • [?] Moving task archive to git

    • evprep

      • [9] evprep: space invaders event flag

        • which flag location: bit 3, parameter in pipeline needs to be turned on
        • evtool should update the gti according to this flag
      • post-processing: frames with many counts?
    • BTI/GTI:
      • PDF for CalPV linked on wiki page (done?)

      • [7] one PDF per eRASS in wiki

    • ermldet

      • [8] ca. 2% of sources have positional error 0, some also have count error 0

        • can't be fixed quickly, may need replacement of algorithm
      • Johan's proposal? Georg to discuss with Johan (long timescale project)
    • Time dependent RMFs/ARFs (srctool?)

      • [8] RMF

      • [7] ARF

    • [6] Background analysis: projecting closed data onto the sky (XMM SAS) -> 6

    • [4] ersensmap: implement pointing PSF

    • ftfindhotpix: test and update algorithm

    • [1] Light leak in eSASS/pipeline?

    • [1] DAtool: filtering catalogs (separate from DAtool)? -> 1

    • DAtool
      • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
      • move simple copying request to Python / should everything be moved to Python? - ongoing
      • DAtool / eROSE interface: requires change in frontend

Previous meeting:

  • eSASS Q&A

  • Open actions from previous developers meetings
    • Act20200929/1_all make sure html task descriptions are current and consistent with user release
      • updates by Jeremy and Ian (evtool, expmap, catprep)
      • still missing: Hermann (radec2xy), Georg (detection + evatt)
    • Act20201006/1_SF clean event file description, update source list description
  • Preprocessing / NRTA
    • Noise/offset maps?
    • TEL chain trigger files / HK file versions

  • eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page

    • eRASS1 code updates/code freeze (Ian's email)

    • expmap: parallel version? Jeremy <--> Ian

      • still under investigation, patch for 947 may be needed
    • catprep: making number of bands variable (1..5) - in progress
    • detection task updates (ongoing, perhaps ready for 947)
      • clobber option
      • improved RADEC_ERR, upper/lower count rate errors
    • Pipeline tasks (shadow developer/pipeline operator): eropipe, proc, prexec, telstage, dalist, rmvers
    • Recent task updates:
      • evprep-3.18, evprep-3.11.3, erolib-2.39, tel-1.24
  • Pipeline / ongoing processing

    • processing wiki page

      • Processing artefacts page rearranged by Chandreyee
    • Computing resources for eSASS pipeline/ground segment vs. heg general
      • NFS server issues
      • Splitting, taring srctool output files, flux threshold
      • Locate command?
    • 947 test runs
      • eRASS1 (first 100 eDays) are in archive

        • HiPS (eRASS1/2?), ProcStat (ok), DAtool

          • DAtool (947): ImageTot -> Image, energy bands: 0-9, a-d

        • Astrometry?
    • eRASS3 (946) exposure deficit
      • caused (in part) by incomplete/not-current HK file versions?
      • telgti issues at border of planning files (fixed in tel-1.24)
      • 46008 and 46085 rerun in eRASS3 946 processing
      • similar exposure deficits in eRASS1/2
    • Missing soft sources (Frank)
      • Ian + Chandreyee will investigate
    • bad pixels in caldb (TM1/2/3/7)
      • TM7: bti for about two eROdays days, many (ca. 50) bad pixels (onboard+eSASS?)
    • Temperature dependence in task energy (HK access)
      • should go in 947 pipeline
      • Konrad will prove list of eROday to Hermann - HB will run eROdays in 947 pipeline!!
    • Zip pattern analysis
      • updating badpixel / bti in caldb
    • Discrepancies in source list of demo script runs / ermldet uninitialized variables
      • Still different results of same binaries on different machines
      • Different from pervious problem, all sources are affected (parameter are sightly different) - high priority
      • Confirm whether ermldet patch changes the eFEDS source list (Teng?)
    • pipeline (eropipe2) machine utilization
      • potential improvements
    • MIP threshold increase (get threshold from HK)
      • evprep-3.18/3.11.3
    • evprep summary reports
      • will be updated by Chandreyee to check TM5/7 against successive eROdays
    • HiPS / ds53 issues / processing speed
      • migrating to He4ero?
  • s/c clock drift / clock resets
  • Longterm activities

    • [?] Moving to git

    • expmap

      • [10] Parallel version

    • evprep

      • [9] evprep: space invaders event flag

        • which flag location: bit 3, parameter in pipeline needs to be turned on
        • evtool should update the gti according to this flag
      • post-processing: frames with many counts?
    • BTI/GTI:
      • PDF for CalPV linked on wiki page (done?)

      • [7] one PDF per eRASS in wiki

    • ermldet

      • [8] ca. 2% of sources have positional error 0, some also have count error 0

        • can't be fixed quickly, may need replacement of algorithm
      • Johan's proposal? Georg to discuss with Johan (long timescale project)
    • Time dependent RMFs/ARFs (srctool?)

      • [8] RMF

      • [7] ARF

    • [6] Background analysis: projecting closed data onto the sky (XMM SAS) -> 6

    • [4] ersensmap: implement pointing PSF

    • ftfindhotpix: test and update algorithm

    • [1] Light leak in eSASS/pipeline?

    • [1] DAtool: filtering catalogs (separate from DAtool)? -> 1

  • DAtool
    • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
    • move simple copying request to Python / should everything be moved to Python? - ongoing
    • DAtool / eROSE interface: requires change in frontend

Describe eSASSdevel_210208 here.

EROSITAwiki: eSASSdevel_210209 (last edited 2021-03-02 08:46:10 by HermannBrunner)