EROSITAwiki: Diff for "eSASSdevel_210216"

Weekly eSASS developers meeting

Start of meeting: 2021-02-16 12:30 <<<< meeting starts earlier this time and will end before 14:00

Zoom link: https://zoom.us/j/97439409844?pwd=YXVkK0hBMnpybm5tL2VyYTh2Qld2UT09

Participants: Ian Stewart, Hermann Brunner, Miriam Ramos, Alain Gueguen, Konrad Dennerl, Mara Salvato, Sabine Osterhage, Susanne Friedrich, Jeremy Sanders. Chandeyee Maitra, Andrea Merloni, Michael Freyberg

• Open actions from previous developers meetings
  • Act20200929/1_all make sure html task descriptions are current and consistent with user release
    • updates by Jeremy and Ian (evtool, expmap, catprep)
    • still missing: Hermann (radec2xy), Georg (detection + evatt)
  • Act20201006/1_SF clean event file description, update source list description
• Preprocessing / NRTA

• eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page

  • eRASS1/2 947 code updates/code freeze
    • perhaps small evprep updates
    • expmap:
      • parallel version (7 cores)
      • withinputmaps=yes
      • testing?
    • catprep: making number of bands variable (1..5) - in progress
    • detection task updates (ongoing, perhaps ready for 947)
      • positional error / count error 0
      • upper/lower count rate errors - will also be propagated by catprep

      • -> ermldet new task version

    • ftfindhotpix / badpix / caldb access in general
      • accessing highest validity date file vs all overlapping file
      • TMIN/TMAX columns not use in current version
      • Ian will make ero_cifsl return files in decending validity start date order
    • Temperature dependent energy calibration
      • Konrad will provide update of task energy (will require new caldb file)
      • Pattern update: read badpix start and eion)
  • fftw version (Alain)
    • different ermldet results on old/new machinnd times
  • srctool vs. apetool exposure (help desk questes may be related to fftw version (results with v3.3.9 are ok)
    • v3.3.8 in current user release should be upgraded - should be documented (eSASSnews)
    • Docker version needs to be updated
    • Check version in pipeline eSASS
  • eSASSuser on Ubuntu 20 - after fftw issue resolved
    • eropipe, proc, prexec - Alain
    • telstage, dalist, rmvers - Miriam

    • test environment, current task versions -> eSASSdevel/task archive

  • Recent task updates:

    • erolib-2.40, evtool-2.19, evprep-3.19

• Pipeline / ongoing processing

  • processing wiki page

  • performance issues?
    • replacement for get_dir routine?
  • 947 test runs
    • eRASS1/2, eRASS:2 (first 100 eDays) are in archive

      • HiPS (eRASS1/2), DAtool, ProcStat (ok)

        • DAtool (947): ImageTot -> Image, energy bands: 0-9, a-d for e?01..., e?02..., s?02...

        • HiPS energy bands? stay with the same bands for now (more bands perhaps later)
      • Astrometry?
  • eRASS2/3 946 exposure deficit
    • use latest HK file versions + tel-1.24 (telgti border issues)
    • fixed in 947 / no full 946 rerun of affected data planned
    • Problem might still exist in recently processed data? e.g., eROday 46283, HB will have look
    • tel-1.24 not yet updated
  • Missing soft sources (Frank)
    • pre-946 processing versions contained CalPV test scans
    • confirmation: rerun test scans in 946 pipeline (or 001?)
    • Survey scans on Dec 8 and Dec 10 should not be part of eRASS1
    • em00 files end of Nov. cover part of LMC. These eDays should be rerun in 001 pipeline
  • TM7 bad pixels
    • caldb was updated, eDays 46199+ restarted
    • data look clean now
  • 949 TEL chain
    • for Konrad's temperature dependent energy calibraton
    • run TEL chain up until pattern
    • CalPV + recent data
• Data access
  • eRASS event files / images
    • FWC data

• EDR
  • Web pages, documentation
  • Validation of CalPV/c001 data: guidlines by eSASS team
• DAtool / ProcSTat
  • access restrictions for data request logs + project descriptions (discussion ongoing - details tbd)
  • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
  • move simple copying request to Python / should everything be moved to Python? - ongoing
  • DAtool / eROSE : requires change in frontend
• Longterm activities

  • [?] Moving task archive to git

  • evprep

    • [9] evprep: space invaders event flag

      • evtool should update the gti according to this flag
      • post-processing: frames with many counts?
    • BTI/GTI:
      • PDF for CalPV linked on wiki page (done?)

      • [7] one PDF per eRASS in wiki

  • ermldet

    • [8] ca. 2% of sources have positional error 0, some also have count error 0

      • can't be fixed quickly, may need replacement of algorithm
      • Johan's proposal? Georg to discuss with Johan (long timescale project)
  • Time dependent RMFs/ARFs (srctool?)

    • [8] RMF

    • [7] ARF

  • [6] Background analysis: projecting closed data onto the sky (XMM SAS)

  • [4] ersensmap: implement pointing PSF

  • [3] ftfindhotpix: test and update algorithm

  • [1] Light leak in eSASS/pipeline?

  • [1] DAtool: filtering catalogs (separate from DAtool)?

Previous meeting:

• Open actions from previous developers meetings
  • Act20200929/1_all make sure html task descriptions are current and consistent with user release
    • updates by Jeremy and Ian (evtool, expmap, catprep)
    • still missing: Hermann (radec2xy), Georg (detection + evatt)
  • Act20201006/1_SF clean event file description, u pdate source list description
• Preprocessing / NRTA
  • Noise/offset maps?

• eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page

  • eRASS1/947 code updates/code freeze (Ian's email)
    • perhaps small evprep updates
  • expmap:
    • parallel version (7 cores)
    • second order parallelization (by GTI)
    • withinputmaps=yes
    • testing?
  • catprep: making number of bands variable (1..5) - in progress
  • detection task updates (ongoing, perhaps ready for 947)
    • clobber option
    • improved RADEC_ERR, upper/lower count rate errors
  • Pipeline tasks (shadow developer/pipeline operator)
    • eropipe, proc, prexec - Alain
    • telstage, dalist, rmvers - Miriam

    • test environment, current task versions -> eSASSdevel/task archive

  • Recent task updates:
    • expmap: v2.1.1-3, v2.2, v1.33.4
• Pipeline / ongoing processing

  • processing wiki page

  • performance issues?
    • replacement for get_dir routine?
    • ds53 speed issues?
  • 947 test runs
    • eRASS1/2, eRASS:2 (first 100 eDays) are in archive

      • HiPS (eRASS1/2), DAtool, ProcStat (ok)

        • DAtool (947): ImageTot -> Image, energy bands: 0-9, a-d for e?01..., e?02..., s?02...

        • HiPS energy bands?
      • Astrometry?
  • eRASS3 (946) exposure deficit
    • caused (in part) by incomplete/not-current HK file versions - fixed
    • telgti issues at border of planning files (fixed in tel-1.24)
    • 46008 and 46085 rerun in eRASS3 946 processing - HiPS maps?
    • similar exposure deficits in eRASS1/2
  • Missing soft sources (Frank)
    • Ian, Chandreyee, Frank investigated
      • probable cause: previous processings contained CalPV test scans
      • confirmation: rerun test scans in 946 pipeline (or 001?)
  • bad pixels
    • TM7:
      • 42 bad pixels in caldb (onboard vs. caldb?)
        • onboard: 40 bad pixels
      • How to proceed with eDay 46199?
        • Susanne will set vsd to beginning of eROday 46199
  • Temperature dependence in task energy (HK access)
    • should go in 947 pipeline
    • Konrad proved list of eROdays to Hermann - TEL_949 chains currently running
  • Zip pattern analysis
    • Susnanne + Chandreyee will discuss
  • Discrepancies in source list of demo script runs / ermldet uninitialized variables
    • Still different results of same binaries on different machines
    • Different from pervious problem, all sources are affected (parameter are sightly different) - high priority
    • Confirm whether ermldet patch changes the eFEDS source list (Teng?)
  • evprep summary reports
    • will be updated by Chandreyee to check TM5/7 against successive eROdays
  • EDR
    • Web pages, documentation
    • Validation of CalPV/c001 data: guidlines by eSASS team

      • https://wiki.mpe.mpg.de/eRosita/guide_validationEDR

• DAtool
  • access restrictions sto data request logs + project descriptions (discussen ongoing- details tbd)
  • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
  • move simple copying request to Python / should everything be moved to Python? - ongoing
  • DAtool / eROSE interface: requires change in frontend

• Longterm activities

  • [?] Moving task archive to git

  • evprep

    • [9] evprep: space invaders event flag

      • which flag location: bit 3, parameter in pipeline needs to be turned on
      • evtool should update the gti according to this flag
    • post-processing: frames with many counts?
  • BTI/GTI:
    • PDF for CalPV linked on wiki page (done?)

    • [7] one PDF per eRASS in wiki

  • ermldet

    • [8] ca. 2% of sources have positional error 0, some also have count error 0

      • can't be fixed quickly, may need replacement of algorithm
    • Johan's proposal? Georg to discuss with Johan (long timescale project)
  • Time dependent RMFs/ARFs (srctool?)

    • [8] RMF

    • [7] ARF

  • [6] Background analysis: projecting closed data onto the sky (XMM SAS) -> 6

  • [4] ersensmap: implement pointing PSF

  • ftfindhotpix: test and update algorithm

  • [1] Light leak in eSASS/pipeline?

  • [1] DAtool: filtering catalogs (separate from DAtool)? -> 1