EROSITAwiki: Diff for "eSASSdevel_210223"

Weekly eSASS developers meeting

Start of meeting: 2021-02-23 13:30 (usual time again)

Zoom link: https://zoom.us/j/97439409844?pwd=YXVkK0hBMnpybm5tL2VyYTh2Qld2UT09

Participants: Ian Stewart, Hermann Brunner, Miriam Ramos, Alain Gueguen, Konrad Dennerl, Mara Salvato, Sabine Osterhage, Susanne Friedrich, Jeremy Sanders. Chandeyee Maitra, Andrea Merloni, Michael Freyberg

• Open actions from previous developers meetings
  • Act20200929/1_all make sure html task descriptions are current and consistent with user release
    • updates by Jeremy and Ian (evtool, expmap, catprep)
    • still missing: Hermann (radec2xy), Georg (detection + evatt)
  • Act20201006/1_SF clean event file description, update source list description
• Preprocessing / NRTA
• eFEDS catalog paper
  • contributers
  • division of work
  • technical appendices
    • level of detail?
    • copying from wiki?
  • times scales
  • further meetings

• eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page

  • evprep
    • discuss what needs to be updated for 947 proessing
  • expmap
    • withintputmaps=yes
    • new parameter inputmaps
  • catprep
  • makefile flags
  • ermldet
    • positional error / count error 0
      • problem identified but not yet fixed
    • upper/lower count rate errors - needs to be propagated by catprep

    • now works with non-automatic flag: -> ermldet-1.37

  • ftfindhotpix / badpix / caldb access in general
    • ero_cifsl now return files in decending validity start date order (current is first)
    • now evaluates TMIN/TMAX / needs to be consistent with pattern (Ian/Konrad will discuss offline)
    • event list extension seems to have fewer badpix entries than caldb (TM7) - Susanne
  • Temperature dependent energy calibration
    • New energy caldb file, calvers.indx needed for eSASSdevel/001/946/947 (make sure that this works)
  • fftw version (Alain)
    • different ermldet results on old/new machines

    • upgrading -> 3.3.9, also Docker

    • still different results between xenial and eSASSuser
    • pipeline eSASS/he*ero machines: fftw taken from environment
      • probably fftw-3.3.7 in pipeline eSASS
    • testing still ongoing
  • Pipeline tasks (shadow developer/pipeline operator)
    • eropipe, proc, prexec - Alain
    • telstage, dalist, rmvers - Miriam, ?
    • test environment, current task versions
  • Recent task updates:
    • gtcal-1.13
    • ftfindhotpix-1.8
    • expmap-2.3
    • ermldet-1.37
• Pipeline / ongoing processing

  • processing wiki page

  • 947 test runs
    • GL's boresight analysis
    • missing polar sky tiles (087156, 093156)
    • Timescale for full 947 TEL/EXP chain runs?
      • Required update/fixes?

    • DAtool / ProcStat?

  • 001 rerun of 082156 (eDays 43546-56)
  • eRASS2/3 946 exposure deficit
    • use latest HK file versions + tel-1.24 (telgti border issues)
    • Sky tile 46008 + 46085, 946 processing sine Feb. 17
  • TM3/TM4 badpix
  • TEL_949 chain
    • test data for for temperature dependent energy calibration

• All-sky data access / SciServer

• EDR
  • Web pages, documentation
  • Validation of CalPV/c001 data: guidlines by eSASS team

• DAtool / ProcSTat
  • access restrictions for data request logs + project descriptions (discussion ongoing - details tbd)
  • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
  • move simple copying request to Python - ongoing
• Longterm activities

  • [?] Moving task archive to git

  • evprep

    • [9] evprep: space invaders event flag

      • evtool should update the gti according to this flag
      • post-processing: frames with many counts?
    • BTI/GTI:
      • PDF for CalPV linked on wiki page (done?)

      • [7] one PDF per eRASS in wiki

  • ermldet

    • [8] ca. 2% of sources have positional error 0, some also have count error 0

      • can't be fixed quickly, may need replacement of algorithm
      • Johan's proposal? Georg to discuss with Johan (long timescale project)
  • Time dependent RMFs/ARFs (srctool?)

    • [8] RMF

    • [7] ARF

  • [6] Background analysis: projecting closed data onto the sky (XMM SAS)

  • [4] ersensmap: implement pointing PSF

  • [3] ftfindhotpix: test and update algorithm

  • [1] Light leak in eSASS/pipeline?

  • [1] DAtool: filtering catalogs (separate from DAtool)?

Previous meeting:

• Open actions from previous developers meetings
  • Act20200929/1_all make sure html task descriptions are current and consistent with user release
    • updates by Jeremy and Ian (evtool, expmap, catprep)
    • still missing: Hermann (radec2xy), Georg (detection + evatt)
  • Act20201006/1_SF clean event file description, update source list description
• Preprocessing / NRTA

• eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page

  • eRASS1/2 947 code updates/code freeze
    • perhaps small evprep updates
    • expmap:
      • parallel version (7 cores)
      • withinputmaps=yes
      • testing?
    • catprep: making number of bands variable (1..5) - in progress
    • detection task updates (ongoing, perhaps ready for 947)
      • positional error / count error 0
      • upper/lower count rate errors - will also be propagated by catprep

      • -> ermldet new task version

    • ftfindhotpix / badpix / caldb access in general
      • accessing highest validity date file vs all overlapping file
      • TMIN/TMAX columns not use in current version
      • Ian will make ero_cifsl return files in decending validity start date order
    • Temperature dependent energy calibration
      • Konrad will provide update of task energy (will require new caldb file)
      • Pattern update: read badpix start and eion)
  • fftw version (Alain)
    • different ermldet results on old/new machines
  • srctool vs. apetool exposure (help desk questes may be related to fftw version (results with v3.3.9 are ok)
    • v3.3.8 in current user release should be upgraded - should be documented (eSASSnews)
    • Docker version needs to be updated
    • Check version in pipeline eSASS
  • eSASSuser on Ubuntu 20 - after fftw issue resolved
    • eropipe, proc, prexec - Alain
    • telstage, dalist, rmvers - Miriam

    • test environment, current task versions -> eSASSdevel/task archive

  • Recent task updates:

    • erolib-2.40, evtool-2.19, evprep-3.19

• Pipeline / ongoing processing

  • processing wiki page

  • performance issues?
    • replacement for get_dir routine?
  • 947 test runs
    • eRASS1/2, eRASS:2 (first 100 eDays) are in archive

      • HiPS (eRASS1/2), DAtool, ProcStat (ok)

        • DAtool (947): ImageTot -> Image, energy bands: 0-9, a-d for e?01..., e?02..., s?02...

        • HiPS energy bands? stay with the same bands for now (more bands perhaps later)
      • Astrometry?
  • eRASS2/3 946 exposure deficit
    • use latest HK file versions + tel-1.24 (telgti border issues)
    • fixed in 947 / no full 946 rerun of affected data planned
    • Problem might still exist in recently processed data? e.g., eROday 46283, HB will have look
    • tel-1.24 not yet updated
  • Missing soft sources (Frank)
    • pre-946 processing versions contained CalPV test scans
    • confirmation: rerun test scans in 946 pipeline (or 001?)
    • Survey scans on Dec 8 and Dec 10 should not be part of eRASS1
    • em00 files end of Nov. cover part of LMC. These eDays should be rerun in 001 pipeline
  • TM7 bad pixels
    • caldb was updated, eDays 46199+ restarted
    • data look clean now
  • 949 TEL chain
    • for Konrad's temperature dependent energy calibraton
    • run TEL chain up until pattern
    • CalPV + recent data
• Data access
  • eRASS event files / images
    • FWC data

• EDR
  • Web pages, documentation
  • Validation of CalPV/c001 data: guidlines by eSASS team
• DAtool / ProcSTat
  • access restrictions for data request logs + project descriptions (discussion ongoing - details tbd)
  • providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
  • move simple copying request to Python / should everything be moved to Python? - ongoing
  • DAtool / eROSE : requires change in frontend
• Longterm activities

  • [?] Moving task archive to git

  • evprep

    • [9] evprep: space invaders event flag

      • evtool should update the gti according to this flag
      • post-processing: frames with many counts?
    • BTI/GTI:
      • PDF for CalPV linked on wiki page (done?)

      • [7] one PDF per eRASS in wiki

  • ermldet

    • [8] ca. 2% of sources have positional error 0, some also have count error 0

      • can't be fixed quickly, may need replacement of algorithm
      • Johan's proposal? Georg to discuss with Johan (long timescale project)
  • Time dependent RMFs/ARFs (srctool?)

    • [8] RMF

    • [7] ARF

  • [6] Background analysis: projecting closed data onto the sky (XMM SAS)

  • [4] ersensmap: implement pointing PSF

  • [3] ftfindhotpix: test and update algorithm

  • [1] Light leak in eSASS/pipeline?

  • [1] DAtool: filtering catalogs (separate from DAtool)?