eSASSdevel_210921 - EROSITAwiki

Weekly eSASS developers Zoom meeting

Start of meeting: 2021-09-21 13:30

Zoom link: https://zoom.us/j/97439409844?pwd=YXVkK0hBMnpybm5tL2VyYTh2Qld2UT09

Participants: Hermann Brunner, Georg Lamer, Michael Freyberg, Felix Schmidt, Teng Liu, Sabine Osterhage, Mara Salvato, Andrea Merloni, Jeremy Sanders

Preprocessing / NRTA
Pipeline / ongoing processing
- processing wiki page
- 947
  - eRASS1 100 eROday DET chains
    - source catalogs
      - missing hardness ratios
        mixed processing versions?
        less cluster detections than in previous versions!
        confirmed by Georg and Teng
        SCUT/ECUT?
      - vs eFEDS ero100 vs eFEDS
      - announcing merged catalog to full team (erosita-members)
    - erbackmap in (almost) empty fields
      - hanging erbackmap runs (killed after >500minexecution time): sky tiles 139057, 037165
        run eFEDS CalPV in 947 pipeline
        running Teng's pipeline on a shallower version of eFEDS
        comparing 946/947 eRASS1 clusters in eFEDS field
        vary task parameters in selected fields
        check pipeline script
      - announcing merged catalog to full team (erosita-members)
    - erbackmap in (almost) empty fields
      - hanging erbackmap runs (killed after >500min execution time): sky tiles 139057, 037165
    - pipeline issues / log file access
    - SOU chains / srctool
  - CalPV data
- 946
- 001 (RXJ 1605)
  - TM5/7 are missing (caldb version mismatch)
eSASS tasks eSASSdevel page / Error tracking page
- prexec
  - now prevents 'hanging' START_... parameter files
- param
  - how to handle concurrent log file accesses
    - finetuning wait time?
    - one pipeline log file per machine?
- from previous meetings:
  - ermldet
    - Mara: RA/DEC/RADEC_ERR contained -999 (ermlet-1.47?) Julien will report details
      - Georg didn't fin -999 but just null values. Happens only very rarely. <- resolved
  - ftfindhotpix / caldb
    - Konrad's proposal (discarding/renaming flags) - should not break backward compatibility -> after 947 is out
  - evprep
    - Konrad provided scaled threshold map
    - dedicated meeting of Konrad and Ian
    - Susanne should put new threshold maps in
    - will only affect data after 47036 (is in eRASS4) - not need for eRASS:3 reprocessing
    - Update needed to address issue with bad pixel neighbours - could be implemented before eRASS3 947 reprocessing (see previous meeting)
- License files (once again)
  - Correct name of license file
    - rename gpl-2.0.. -> COPYING? plus license.txt -> AUTHORS
  - Jeremy/Alain/HB/Johannes(?)/Bamberg(Sixte?) to discuss and make final decision
  - Jeremy will look how/whether the individual task authors are listed in SIXTE
- New user release
  - Test version by Ian
  - Reporting version number in user release (method by Philipp Weber)
- recent task updates:
  - srctool-1.69
  - prexec-1.10
  - eroio-2.68
  - ermldet-1.50
  - erbackmap-1.21 (22?)
  - param-1.42, 143 (not yet in sw_archive)
HiPS maps
- computing resources
  - Java / FTOOLS
  - Update frequency (already implemented) / rates images made by pipline?
    - exposure and rates maps are run twice per week
DAtool
- server time-outs (causes?)
- Backend
  - delivering 947 products
  - version numbers in file names
- Frontend
  - emails
    - short term: change email text (HB will provide text)
    - delete button
    - full text link under buttons (data / delete)
    - medium term: send second (confirmation) email from backend
  - Request log:
    - links to region files
  - Deleting old (older than 3 months) request direcctorys
- access privileges for www/DataRequests (Harald)
  - DataRequests dir should be group eroproc with no world read access
Script to obtain sky tiles (Miriam)
Longterm activities
- [?] Moving task archive to git
- evprep
  - [9] evprep: space invaders event flag
    - implemented - should be documented in next users release
    - more detailed, quantitative checks regarding improvement of data quality
    - evtool should update the gti according to this flag
      - wider issue: should discarded frames due to flag selection b tracked in GTI?
  - BTI/GTI:
    - [7] one PDF per eRASS in wiki
- [7] PSF normalization
  - caldb vs. shaplet PSF (Antonis)
  - may be necessary to put 1 arcmin vs. 4 armin flux correction into shaplet caldb file (needs testing) -> longterm activity
  - ermldet
    - [3] Johan's proposal? Georg to discuss with Johan (long timescale project)
    - [7] Proposal by Florian: don't use fixed extrction radius but one dependent on flux
  - Time dependent RMFs/ARFs (srctool?)
    - [8] RMF
    - [7] ARF
  - [1] EV_WEIGHT: event based vignetting correction (would require dedicated tests)
  - [6] Background analysis: projecting closed data onto the sky (XMM SAS)
  - [6] ersensmap: implement pointing PSF
  - [3] ftfindhotpix: test and update algorithm
  - [1] Light leak in eSASS/pipeline?
  - [1] DAtool: filtering catalogs?
  - [6] add option to provide larger sky areas (Jeremy's script)

Previous meeting:

Q&A
Preprocessing / NRTA
Pipeline / ongoing processing
- processing wiki page
- 947
  - eRASS1 100 eROday DET chains
    - source catalogs
      - owner flag not set
      - ape tool data written to separate file
      - merged catalog
    - chain processing
      - hanging erbackmap in pathological fields (only exposure in corners) - ca. 10 sky tiles
      - hanging after apetool forced photometry runs - ca. 20 sky tiles
    - dividing srctool products into multiple files
      - Hermann: divide by eRASS (in SOU_947.PAR) <- agreed
      - Jeremy: tar all files of one source in srctool <- agreed
      - Alain: DAtsool backend needs to be adjusted
      - look at suggestion in 947 wiki page and make suggestion at data meeting
    - start eROday of eRASS1
      - start on Dec. 11
- 946
  - processed TM4 data
    - everything before threshod change is processed, will be continued when Ian is back
- Cal pointings after orbit correction
  - RXJ 1605 (001)
    - TM5/7 errors
      - incompatible algos cal file?
      - will be resolved when Ian and Susanne re back
  - ART observations (946)
    - planning & PI file updated / running
- disk space / copying
  - declination range 024
  - group read access in archive?
eSASS tasks eSASSdevel page / Error tracking page
- from previous meetings:
  - ermldet
    - Mara: RA/DEC/RADEC_ERR contained -999 (ermlet-1.47?) Julien will report details
      - Georg didn't fin -999 but just null values. Happens only very rarely. <- resolved
  - ftfindhotpix / caldb
    - Konrad's proposal (discarding/renaming flags) - should not break backward compatibility -> after 947 is out
  - evprep
    - Konrad provided scaled threshold map
    - dedicated meeting of Konrad and Ian
    - Susanne should put new threshold maps in
    - will only affect data after 47036 (is in eRASS4) - not need for eRASS:3 reprocessing
    - Update needed to address issue with bad pixel neighbours - could be implemented before eRASS3 947 reprocessing (see previous meeting)
  - License files (once again)
    - Correct name of license file?
    - Jeremy/Alain/HB/Johannes(?)/Bamberg(Sixte?) to discuss and make final decision
    - Jeremy will look how/whether the individual task authors are listed in SIXT
  - New user release
    - Test version by Ian
    - Reporting version number in user release (method by Philipp Weber)
  - recent task updates:
    - evprep-3.44
    - erolib-1.66
    - catprep-2.8.3
    - erbackmap-??
user questions
- helpdesk
- forum
  - Nico's question
  - can one subscribe to forums?
DAtool
- Backend
  - delivering 947 products
  - version numbers in file names
  - speed improvements
- Frontend
  - emails
    - short term: change email text
    - delete button
    - full text link under buttons (data / delete)
    - medium term: send second (confirmation) email from backend
  - Request log:
    - links to region files
- access privileges for www/DataRequests (remind Harald)
Longterm activities
- [?] Moving task archive to git
- evprep
  - [9] evprep: space invaders event flag
    - implemented - should be documented in next users release
    - more detailed, quantitative checks regarding improvement of data quality
    - evtool should update the gti according to this flag
      - wider issue: should discarded frames due to flag selection b tracked in GTI?
  - BTI/GTI:
    - [7] one PDF per eRASS in wiki
- [7] PSF normalization
  - caldb vs. shaplet PSF (Antonis)
  - may be necessary to put 1 arcmin vs. 4 armin flux correction into shaplet caldb file (needs testing) -> longterm activity
  - ermldet
    - [3] Johan's proposal? Georg to discuss with Johan (long timescale project)
    - [7] Proposal by Florian: don't use fixed extrction radius but one dependent on flux
  - Time dependent RMFs/ARFs (srctool?)
    - [8] RMF
    - [7] ARF
  - [1] EV_WEIGHT: event based vignetting correction (would require dedicated tests)
  - [6] Background analysis: projecting closed data onto the sky (XMM SAS)
  - [6] ersensmap: implement pointing PSF
  - [3] ftfindhotpix: test and update algorithm
  - [1] Light leak in eSASS/pipeline?
  - [1] DAtool: filtering catalogs?
  - [6] add option to provide larger sky areas (Jeremy's script)

EROSITAwiki: eSASSdevel_210921 (last edited 2021-09-28 09:35:53 by JeremySanders)