Weekly eSASS developers Zoom meeting
Start of meeting: 2021-09-28 13:30
Zoom link: https://zoom.us/j/97439409844?pwd=YXVkK0hBMnpybm5tL2VyYTh2Qld2UT09
Participants: Hermann Brunner, Georg Lamer, Michael Freyberg, Felix Schmidt, Teng Liu, Sabine Osterhage, Mara Salvato, Andrea Merloni
- Preprocessing / NRTA
- Pipeline / ongoing processing
- 947
- Processing
- eFEDS (CalPV) TEL/EXP/DET_947
- Check results / TM6 time shift corrected?
- timing is correct now
- Check results / TM6 time shift corrected?
- eRASS1/2/3 100 eROday test TEL_947 chains reprocessed
processed with evprep-3.43 / evprep-3.44 failed in EXP chain (expmerge) -> Ian
- some hanging chains, ok when reprocessed
- starting full-sky TEL_947 chains?
- eFEDS (CalPV) TEL/EXP/DET_947
- Analysis / improvement of 947 detection results
- Teng's/Georg's presentations at Data/Cal meeting
- ermldet testing / update? (Georg)
- no issue with ermldet-1.50 PSF/extent fft convolution
- Comparison 947 eFEDS (CalPV) to Teng's eFEDS detection?
- extent likelihoods somewhat lower, can in part be recovered be lowering threshold
- more tests needed / 947 det. worked better (improved with repsect to Tengs's eFEDS cat?) for eFEDS then for (shallower) eRASS1
- extent likelihoods somewhat lower, can in part be recovered be lowering threshold
- What to do next:
- check attitude in eFEDS 947 and Jeremy's test (merged attitude - evtool issue?)
- ermldet-1.50 - Georg will analyse further
- erbackmap: extend extraction radius?
Teng's slide c947 vs c001
- From previous meeting:
- missing hardness ratios - fixed
- ermldet slope parameter (extlike_slope): also comma missing in par file?
- hanging erbackmap - test outside of pipeline
- Processing
- 946
eSASS tasks eSASSdevel page / Error tracking page
- expmap
- issues when merging observations (jeremy)
- attitude problem when merging with evtool?
- issues when merging observations (jeremy)
- ermldet
- fft PSf/extent convolution (Georg)
- fft convolution seems ok, other ermldet issues affecting extent detection still under investigation
- fft PSf/extent convolution (Georg)
- ersensmap in pointings
- fixing crashes?
- remove from PDET_947? (pointings / field scans)
- add to error page
- proc/prexec
- hanging start files
- some code changes but still issues (HB)
- hanging start files
- License files, etcl.
- Leave text in code as is
- Don't change anything in current user release
- BUT: LICENCE.TXT is misspelled? NO correct UK spelling!
- For next user release / eSASSdevel
gpl-2.0.txt --> COPYING, LICENCE.TXT --> AUTHORS
- From previous meeting:
- ftfindhotpix / caldb
Konrad's proposal (discarding/renaming flags) - should not break backward compatibility -> after 947 is out
- evprep
- Konrad provided scaled threshold map
- dedicated meeting of Konrad and Ian
- Susanne should put new threshold maps in
- will only affect data after 47036 (is in eRASS4) - not need for eRASS:3 reprocessing
- Update needed to address issue with bad pixel neighbours - could be implemented before eRASS3 947 reprocessing (see previous meeting)
- ftfindhotpix / caldb
- recent task updates
- expmap
helpdesk / user questions
- Beate's request / empty directory
- Werner: issues in WG chair email?
- DAtool
- server time-outs (causes?)
- call backend assynchronously (don't wait)
HB email with example to Sabine & Alain
- sending final confirmation email from backend
- Adjusting email texts
- call backend assynchronously (don't wait)
- Backend: delivering 947 detection products (catalogs, auxiliary)
- testing
- Frontend/emails: full link text under buttons
- Users without expiration data?
Empty DataRequests directories?
- access privileges for www/DataRequests (Harald)
DataRequests dir should be group eroproc with no world read access
- not yet fully consistent but should be ok
- server time-outs (causes?)
- Longterm activities
[?] Moving task archive to git
- evprep
[9] evprep: space invaders event flag
- implemented - should be documented in next users release
- more detailed, quantitative checks regarding improvement of data quality
- evtool should update the gti according to this flag
- wider issue: should discarded frames due to flag selection b tracked in GTI?
- BTI/GTI:
[7] one PDF per eRASS in wiki
[7] PSF normalization
- caldb vs. shaplet PSF (Antonis)
may be necessary to put 1 arcmin vs. 4 armin flux correction into shaplet caldb file (needs testing) -> longterm activity
- ermldet
[3] Johan's proposal? Georg to discuss with Johan (long timescale project)
[7] Proposal by Florian: don't use fixed extrction radius but one dependent on flux
- Time dependent RMFs/ARFs (srctool?)
[8] RMF
[7] ARF
[1] EV_WEIGHT: event based vignetting correction (would require dedicated tests)
[6] Background analysis: projecting closed data onto the sky (XMM SAS)
[6] ersensmap: implement pointing PSF
[3] ftfindhotpix: test and update algorithm
[1] Light leak in eSASS/pipeline?
[1] DAtool: filtering catalogs?
[6] add option to provide larger sky areas (Jeremy's script)
Previous meeting:
- Preprocessing / NRTA
- Pipeline / ongoing processing
- 947
- eRASS1 100 eROday DET chains
- source catalogs
- missing hardness ratios
- mixed processing versions?
- less cluster detections than in previous versions!
- confirmed by Georg and Teng
- SCUT/ECUT?
vs eFEDS ero100 vs eFEDS
- announcing merged catalog to full team (erosita-members)
- missing hardness ratios
- erbackmap in (almost) empty fields
hanging erbackmap runs (killed after >500minexecution time): sky tiles 139057, 037165
- run eFEDS CalPV in 947 pipeline
- running Teng's pipeline on a shallower version of eFEDS
- comparing 946/947 eRASS1 clusters in eFEDS field
- vary task parameters in selected fields
- check pipeline script
- announcing merged catalog to full team (erosita-members)
- erbackmap in (almost) empty fields
hanging erbackmap runs (killed after >500min execution time): sky tiles 139057, 037165
- pipeline issues / log file access
- SOU chains / srctool
- source catalogs
- CalPV data
- eRASS1 100 eROday DET chains
- 946
- 001 (RXJ 1605)
- TM5/7 are missing (caldb version mismatch)
eSASS tasks eSASSdevel page / Error tracking page
- prexec
- now prevents 'hanging' START_... parameter files
- param
- how to handle concurrent log file accesses
- finetuning wait time?
- one pipeline log file per machine?
- how to handle concurrent log file accesses
- from previous meetings:
- ermldet
- Mara: RA/DEC/RADEC_ERR contained -999 (ermlet-1.47?) Julien will report details
Georg didn't fin -999 but just null values. Happens only very rarely. <- resolved
- Mara: RA/DEC/RADEC_ERR contained -999 (ermlet-1.47?) Julien will report details
- ftfindhotpix / caldb
Konrad's proposal (discarding/renaming flags) - should not break backward compatibility -> after 947 is out
- evprep
- Konrad provided scaled threshold map
- dedicated meeting of Konrad and Ian
- Susanne should put new threshold maps in
- will only affect data after 47036 (is in eRASS4) - not need for eRASS:3 reprocessing
- Update needed to address issue with bad pixel neighbours - could be implemented before eRASS3 947 reprocessing (see previous meeting)
- ermldet
- License files (once again)
- Correct name of license file
rename gpl-2.0.. -> COPYING? plus license.txt -> AUTHORS
- Jeremy/Alain/HB/Johannes(?)/Bamberg(Sixte?) to discuss and make final decision
- Jeremy will look how/whether the individual task authors are listed in SIXTE
- Correct name of license file
- New user release
- Test version by Ian
- Reporting version number in user release (method by Philipp Weber)
- recent task updates:
- srctool-1.69
- prexec-1.10
- eroio-2.68
- ermldet-1.50
- erbackmap-1.21 (22?)
- param-1.42, 143 (not yet in sw_archive)
- prexec
- HiPS maps
- computing resources
- Java / FTOOLS
- Update frequency (already implemented) / rates images made by pipline?
- exposure and rates maps are run twice per week
- computing resources
- DAtool
- server time-outs (causes?)
- Backend
- delivering 947 products
- version numbers in file names
- Frontend
- emails
- short term: change email text (HB will provide text)
- delete button
- full text link under buttons (data / delete)
- medium term: send second (confirmation) email from backend
- Request log:
- links to region files
- Deleting old (older than 3 months) request direcctorys
- emails
- access privileges for www/DataRequests (Harald)
DataRequests dir should be group eroproc with no world read access
- Script to obtain sky tiles (Miriam)
- Longterm activities
[?] Moving task archive to git
- evprep
[9] evprep: space invaders event flag
- implemented - should be documented in next users release
- more detailed, quantitative checks regarding improvement of data quality
- evtool should update the gti according to this flag
- wider issue: should discarded frames due to flag selection b tracked in GTI?
- BTI/GTI:
[7] one PDF per eRASS in wiki
[7] PSF normalization
- caldb vs. shaplet PSF (Antonis)
may be necessary to put 1 arcmin vs. 4 armin flux correction into shaplet caldb file (needs testing) -> longterm activity
- ermldet
[3] Johan's proposal? Georg to discuss with Johan (long timescale project)
[7] Proposal by Florian: don't use fixed extrction radius but one dependent on flux
- Time dependent RMFs/ARFs (srctool?)
[8] RMF
[7] ARF
[1] EV_WEIGHT: event based vignetting correction (would require dedicated tests)
[6] Background analysis: projecting closed data onto the sky (XMM SAS)
[6] ersensmap: implement pointing PSF
[3] ftfindhotpix: test and update algorithm
[1] Light leak in eSASS/pipeline?
[1] DAtool: filtering catalogs?
[6] add option to provide larger sky areas (Jeremy's script)