Weekly eSASS developers Zoom meeting

Start of meeting: 2021-09-21 13:30

Zoom link: https://zoom.us/j/97439409844?pwd=YXVkK0hBMnpybm5tL2VyYTh2Qld2UT09

Participants: Hermann Brunner, Georg Lamer, Michael Freyberg, Felix Schmidt, Teng Liu, Sabine Osterhage, Mara Salvato, Andrea Merloni, Jeremy Sanders

• Preprocessing / NRTA
• Pipeline / ongoing processing

  • processing wiki page

  • 947
    • eRASS1 100 eROday DET chains
      • source catalogs
        • missing hardness ratios
          • mixed processing versions?
          • less cluster detections than in previous versions!
            • confirmed by Georg and Teng
            • SCUT/ECUT?

        • vs eFEDS ero100 vs eFEDS

        • announcing merged catalog to full team (erosita-members)
      • erbackmap in (almost) empty fields

        • hanging erbackmap runs (killed after >500minexecution time): sky tiles 139057, 037165

          • run eFEDS CalPV in 947 pipeline
          • running Teng's pipeline on a shallower version of eFEDS
          • comparing 946/947 eRASS1 clusters in eFEDS field
          • vary task parameters in selected fields
          • check pipeline script
        • announcing merged catalog to full team (erosita-members)
      • erbackmap in (almost) empty fields

        • hanging erbackmap runs (killed after >500min execution time): sky tiles 139057, 037165

      • pipeline issues / log file access
      • SOU chains / srctool
    • CalPV data
  • 946
  • 001 (RXJ 1605)
    • TM5/7 are missing (caldb version mismatch)

• eSASS tasks eSASSdevel page / Error tracking page

  • prexec
    • now prevents 'hanging' START_... parameter files
  • param
    • how to handle concurrent log file accesses
      • finetuning wait time?
      • one pipeline log file per machine?
  • from previous meetings:
    • ermldet
      • Mara: RA/DEC/RADEC_ERR contained -999 (ermlet-1.47?) Julien will report details

        • Georg didn't fin -999 but just null values. Happens only very rarely. <- resolved

    • ftfindhotpix / caldb

      • Konrad's proposal (discarding/renaming flags) - should not break backward compatibility -> after 947 is out

    • evprep
      • Konrad provided scaled threshold map
      • dedicated meeting of Konrad and Ian
      • Susanne should put new threshold maps in
      • will only affect data after 47036 (is in eRASS4) - not need for eRASS:3 reprocessing
      • Update needed to address issue with bad pixel neighbours - could be implemented before eRASS3 947 reprocessing (see previous meeting)
  • License files (once again)
    • Correct name of license file

      • rename gpl-2.0.. -> COPYING? plus license.txt -> AUTHORS

    • Jeremy/Alain/HB/Johannes(?)/Bamberg(Sixte?) to discuss and make final decision
    • Jeremy will look how/whether the individual task authors are listed in SIXTE
  • New user release
    • Test version by Ian
    • Reporting version number in user release (method by Philipp Weber)
  • recent task updates:
    • srctool-1.69
    • prexec-1.10
    • eroio-2.68
    • ermldet-1.50
    • erbackmap-1.21 (22?)
    • param-1.42, 143 (not yet in sw_archive)
• HiPS maps
  • computing resources
    • Java / FTOOLS
    • Update frequency (already implemented) / rates images made by pipline?
      • exposure and rates maps are run twice per week
• DAtool
  • server time-outs (causes?)
  • Backend
    • delivering 947 products
    • version numbers in file names
  • Frontend
    • emails
      • short term: change email text (HB will provide text)
      • delete button
      • full text link under buttons (data / delete)
      • medium term: send second (confirmation) email from backend
    • Request log:
      • links to region files
    • Deleting old (older than 3 months) request direcctorys
  • access privileges for www/DataRequests (Harald)

    • DataRequests dir should be group eroproc with no world read access

• Script to obtain sky tiles (Miriam)
• Longterm activities

  • [?] Moving task archive to git

  • evprep

    • [9] evprep: space invaders event flag

      • implemented - should be documented in next users release
      • more detailed, quantitative checks regarding improvement of data quality
      • evtool should update the gti according to this flag
        • wider issue: should discarded frames due to flag selection b tracked in GTI?
    • BTI/GTI:

      • [7] one PDF per eRASS in wiki

  • [7] PSF normalization

    • caldb vs. shaplet PSF (Antonis)

    • may be necessary to put 1 arcmin vs. 4 armin flux correction into shaplet caldb file (needs testing) -> longterm activity

    • ermldet

      • [3] Johan's proposal? Georg to discuss with Johan (long timescale project)

      • [7] Proposal by Florian: don't use fixed extrction radius but one dependent on flux

    • Time dependent RMFs/ARFs (srctool?)

      • [8] RMF

      • [7] ARF

    • [1] EV_WEIGHT: event based vignetting correction (would require dedicated tests)

    • [6] Background analysis: projecting closed data onto the sky (XMM SAS)

    • [6] ersensmap: implement pointing PSF

    • [3] ftfindhotpix: test and update algorithm

    • [1] Light leak in eSASS/pipeline?

    • [1] DAtool: filtering catalogs?

    • [6] add option to provide larger sky areas (Jeremy's script)

Previous meeting:

• Q&A

• Preprocessing / NRTA
• Pipeline / ongoing processing

  • processing wiki page

  • 947
    • eRASS1 100 eROday DET chains
      • source catalogs
        • owner flag not set
        • ape tool data written to separate file
        • merged catalog
      • chain processing
        • hanging erbackmap in pathological fields (only exposure in corners) - ca. 10 sky tiles
        • hanging after apetool forced photometry runs - ca. 20 sky tiles
      • dividing srctool products into multiple files

        • Hermann: divide by eRASS (in SOU_947.PAR) <- agreed

        • Jeremy: tar all files of one source in srctool <- agreed

        • Alain: DAtsool backend needs to be adjusted
        • look at suggestion in 947 wiki page and make suggestion at data meeting
      • start eROday of eRASS1
        • start on Dec. 11
  • 946
    • processed TM4 data
      • everything before threshod change is processed, will be continued when Ian is back
  • Cal pointings after orbit correction
    • RXJ 1605 (001)
      • TM5/7 errors
        • incompatible algos cal file?
        • will be resolved when Ian and Susanne re back
    • ART observations (946)

      • planning & PI file updated / running

  • disk space / copying
    • declination range 024
    • group read access in archive?

• eSASS tasks eSASSdevel page / Error tracking page

  • from previous meetings:
    • ermldet
      • Mara: RA/DEC/RADEC_ERR contained -999 (ermlet-1.47?) Julien will report details

        • Georg didn't fin -999 but just null values. Happens only very rarely. <- resolved

    • ftfindhotpix / caldb

      • Konrad's proposal (discarding/renaming flags) - should not break backward compatibility -> after 947 is out

    • evprep
      • Konrad provided scaled threshold map
      • dedicated meeting of Konrad and Ian
      • Susanne should put new threshold maps in
      • will only affect data after 47036 (is in eRASS4) - not need for eRASS:3 reprocessing
      • Update needed to address issue with bad pixel neighbours - could be implemented before eRASS3 947 reprocessing (see previous meeting)
    • License files (once again)
      • Correct name of license file?
      • Jeremy/Alain/HB/Johannes(?)/Bamberg(Sixte?) to discuss and make final decision
      • Jeremy will look how/whether the individual task authors are listed in SIXT
    • New user release
      • Test version by Ian
      • Reporting version number in user release (method by Philipp Weber)
    • recent task updates:
      • evprep-3.44
      • erolib-1.66
      • catprep-2.8.3
      • erbackmap-??
• user questions

  • helpdesk

  • forum
    • Nico's question
    • can one subscribe to forums?
• DAtool
  • Backend
    • delivering 947 products
    • version numbers in file names
    • speed improvements
  • Frontend
    • emails
      • short term: change email text
      • delete button
      • full text link under buttons (data / delete)
      • medium term: send second (confirmation) email from backend
    • Request log:
      • links to region files
  • access privileges for www/DataRequests (remind Harald)
• Longterm activities

  • [?] Moving task archive to git

  • evprep

    • [9] evprep: space invaders event flag

      • implemented - should be documented in next users release
      • more detailed, quantitative checks regarding improvement of data quality
      • evtool should update the gti according to this flag
        • wider issue: should discarded frames due to flag selection b tracked in GTI?
    • BTI/GTI:

      • [7] one PDF per eRASS in wiki

  • [7] PSF normalization

    • caldb vs. shaplet PSF (Antonis)

    • may be necessary to put 1 arcmin vs. 4 armin flux correction into shaplet caldb file (needs testing) -> longterm activity

    • ermldet

      • [3] Johan's proposal? Georg to discuss with Johan (long timescale project)

      • [7] Proposal by Florian: don't use fixed extrction radius but one dependent on flux

    • Time dependent RMFs/ARFs (srctool?)

      • [8] RMF

      • [7] ARF

    • [1] EV_WEIGHT: event based vignetting correction (would require dedicated tests)

    • [6] Background analysis: projecting closed data onto the sky (XMM SAS)

    • [6] ersensmap: implement pointing PSF

    • [3] ftfindhotpix: test and update algorithm

    • [1] Light leak in eSASS/pipeline?

    • [1] DAtool: filtering catalogs?

    • [6] add option to provide larger sky areas (Jeremy's script)