EROSITAwiki: Diff for "eSASSdevel_210209"

Weekly eSASS developers meeting

Start of meeting: 2021-02-09 13:30 (1:30pm)

Zoom link: https://zoom.us/j/97439409844?pwd=YXVkK0hBMnpybm5tL2VyYTh2Qld2UT09

Participants: Ian Stewart, Hermann Brunner, Miriam Ramos, Alain Gueguen, Konrad Dennerl, Mara Salvato, Sabine Osterhage, Susanne Friedrich, Jeremy Sanders. Chandeyee Maitra, Andrea Merloni, Michael Freyberg

• Open actions from previous developers meetings
  • Act20200929/1_all make sure html task descriptions are current and consistent with user release
    • updates by Jeremy and Ian (evtool, expmap, catprep)
    • still missing: Hermann (radec2xy), Georg (detection + evatt)
  • Act20201006/1_SF clean event file description, update source list description
• Preprocessing / NRTA
  • Noise/offset maps?

• eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page

  • eRASS1/947 code updates/code freeze (Ian's email)
    • perhaps small evprep updates
  • expmap:
    • parallel version (7 cores)
    • second order parallelization (by GTI)
    • withinputmaps=yes
    • testing?
  • catprep: making number of bands variable (1..5) - in progress
  • detection task updates (ongoing, perhaps ready for 947)
    • clobber option
    • improved RADEC_ERR, upper/lower count rate errors
  • Pipeline tasks (shadow developer/pipeline operator)
    • eropipe, proc, prexec - Alain
    • telstage, dalist, rmvers - Miriam

    • test environment, current task versions -> eSASSdevel/task archive

  • Recent task updates:
    • expmap: v2.1.1-3, v2.2, v1.33.4
• Pipeline / ongoing processing

  • processing wiki page

  • performance issues?
    • replacement for get_dir routine?
    • ds53 speed issues?
  • 947 test runs
    • eRASS1/2, eRASS:2 (first 100 eDays) are in archive

      • HiPS (eRASS1/2), DAtool, ProcStat (ok)

        • DAtool (947): ImageTot -> Image, energy bands: 0-9, a-d for e?01..., e?02..., s?02...

        • HiPS energy bands?
      • Astrometry?
  • eRASS3 (946) exposure deficit
    • caused (in part) by incomplete/not-current HK file versions - fixed
    • telgti issues at border of planning files (fixed in tel-1.24)
    • 46008 and 46085 rerun in eRASS3 946 processing - HiPS maps?
    • similar exposure deficits in eRASS1/2
  • Missing soft sources (Frank)
    • Ian, Chandreyee, Frank investigated
      • probable cause: previous processings contained CalPV test scans
      • confirmation: rerun test scans in 946 pipeline (or 001?)
  • bad pixels
    • TM7:
      • 42 bad pixels in caldb (onboard vs. caldb?)
        • onboard: 40 bad pixels
      • How to proceed with eDay 46199?
        • Susanne will set vsd to beginning of eROday 46199
  • Temperature dependence in task energy (HK access)
    • should go in 947 pipeline
    • Konrad proved list of eROdays to Hermann - TEL_949 chains currently running
  • Zip pattern analysis
    • Susnanne + Chandreyee will discuss
  • Discrepancies in source list of demo script runs / ermldet uninitialized variables
    • Still different results of same binaries on different machines
    • Different from pervious problem, all sources are affected (parameter are sightly different) - high priority
    • Confirm whether ermldet patch changes the eFEDS source list (Teng?)
  • evprep summary reports
    • will be updated by Chandreyee to check TM5/7 against successive eROdays
  • EDR
    • Web pages, documentation
    • Validation of CalPV/c001 data: guidlines by eSASS team

      • https://wiki.mpe.mpg.de/eRosita/guide_validationEDR

• DAtool
  • access restrictions sto data request logs + project descriptions (discussen ongoing- details tbd)
  • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
  • move simple copying request to Python / should everything be moved to Python? - ongoing
  • DAtool / eROSE interface: requires change in frontend

• Longterm activities

  • [?] Moving task archive to git

  • evprep

    • [9] evprep: space invaders event flag

      • which flag location: bit 3, parameter in pipeline needs to be turned on
      • evtool should update the gti according to this flag
    • post-processing: frames with many counts?
  • BTI/GTI:
    • PDF for CalPV linked on wiki page (done?)

    • [7] one PDF per eRASS in wiki

  • ermldet

    • [8] ca. 2% of sources have positional error 0, some also have count error 0

      • can't be fixed quickly, may need replacement of algorithm
    • Johan's proposal? Georg to discuss with Johan (long timescale project)
  • Time dependent RMFs/ARFs (srctool?)

    • [8] RMF

    • [7] ARF

  • [6] Background analysis: projecting closed data onto the sky (XMM SAS) -> 6

  • [4] ersensmap: implement pointing PSF

  • ftfindhotpix: test and update algorithm

  • [1] Light leak in eSASS/pipeline?

  • [1] DAtool: filtering catalogs (separate from DAtool)? -> 1

  • DAtool
    • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
    • move simple copying request to Python / should everything be moved to Python? - ongoing
    • DAtool / eROSE interface: requires change in frontend

Previous meeting:

• eSASS Q&A

• Open actions from previous developers meetings
  • Act20200929/1_all make sure html task descriptions are current and consistent with user release
    • updates by Jeremy and Ian (evtool, expmap, catprep)
    • still missing: Hermann (radec2xy), Georg (detection + evatt)
  • Act20201006/1_SF clean event file description, update source list description
• Preprocessing / NRTA
  • Noise/offset maps?
  • TEL chain trigger files / HK file versions

• eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page

  • eRASS1 code updates/code freeze (Ian's email)

  • expmap: parallel version? Jeremy <--> Ian

    • still under investigation, patch for 947 may be needed
  • catprep: making number of bands variable (1..5) - in progress
  • detection task updates (ongoing, perhaps ready for 947)
    • clobber option
    • improved RADEC_ERR, upper/lower count rate errors
  • Pipeline tasks (shadow developer/pipeline operator): eropipe, proc, prexec, telstage, dalist, rmvers
  • Recent task updates:
    • evprep-3.18, evprep-3.11.3, erolib-2.39, tel-1.24
• Pipeline / ongoing processing

  • processing wiki page

    • Processing artefacts page rearranged by Chandreyee
  • Computing resources for eSASS pipeline/ground segment vs. heg general
    • NFS server issues
    • Splitting, taring srctool output files, flux threshold
    • Locate command?
  • 947 test runs
    • eRASS1 (first 100 eDays) are in archive

      • HiPS (eRASS1/2?), ProcStat (ok), DAtool

        • DAtool (947): ImageTot -> Image, energy bands: 0-9, a-d

      • Astrometry?
  • eRASS3 (946) exposure deficit
    • caused (in part) by incomplete/not-current HK file versions?
    • telgti issues at border of planning files (fixed in tel-1.24)
    • 46008 and 46085 rerun in eRASS3 946 processing
    • similar exposure deficits in eRASS1/2
  • Missing soft sources (Frank)
    • Ian + Chandreyee will investigate
  • bad pixels in caldb (TM1/2/3/7)
    • TM7: bti for about two eROdays days, many (ca. 50) bad pixels (onboard+eSASS?)
  • Temperature dependence in task energy (HK access)
    • should go in 947 pipeline
    • Konrad will prove list of eROday to Hermann - HB will run eROdays in 947 pipeline!!
  • Zip pattern analysis
    • updating badpixel / bti in caldb
  • Discrepancies in source list of demo script runs / ermldet uninitialized variables
    • Still different results of same binaries on different machines
    • Different from pervious problem, all sources are affected (parameter are sightly different) - high priority
    • Confirm whether ermldet patch changes the eFEDS source list (Teng?)
  • pipeline (eropipe2) machine utilization
    • potential improvements
  • MIP threshold increase (get threshold from HK)
    • evprep-3.18/3.11.3
  • evprep summary reports
    • will be updated by Chandreyee to check TM5/7 against successive eROdays
  • HiPS / ds53 issues / processing speed
    • migrating to He4ero?

• s/c clock drift / clock resets
  • EDR
    • Web pages, documentation
    • Validation of CalPV/c001 data: guidlines by eSASS team

      • https://wiki.mpe.mpg.de/eRosita/guide_validationEDR

• Longterm activities

  • [?] Moving to git

  • expmap

    • [10] Parallel version

  • evprep

    • [9] evprep: space invaders event flag

      • which flag location: bit 3, parameter in pipeline needs to be turned on
      • evtool should update the gti according to this flag
    • post-processing: frames with many counts?
  • BTI/GTI:
    • PDF for CalPV linked on wiki page (done?)

    • [7] one PDF per eRASS in wiki

  • ermldet

    • [8] ca. 2% of sources have positional error 0, some also have count error 0

      • can't be fixed quickly, may need replacement of algorithm
    • Johan's proposal? Georg to discuss with Johan (long timescale project)
  • Time dependent RMFs/ARFs (srctool?)

    • [8] RMF

    • [7] ARF

  • [6] Background analysis: projecting closed data onto the sky (XMM SAS) -> 6

  • [4] ersensmap: implement pointing PSF

  • ftfindhotpix: test and update algorithm

  • [1] Light leak in eSASS/pipeline?

  • [1] DAtool: filtering catalogs (separate from DAtool)? -> 1

• DAtool
  • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
  • move simple copying request to Python / should everything be moved to Python? - ongoing
  • DAtool / eROSE interface: requires change in frontend

Describe eSASSdevel_210208 here.