#acl EroCatGroup:read,write,revert All:read == Weekly eSASS developers Zoom meeting == '''Start of meeting: 2021-09-21 13:30''' Zoom link: https://zoom.us/j/97439409844?pwd=YXVkK0hBMnpybm5tL2VyYTh2Qld2UT09 Participants: Hermann Brunner, Georg Lamer, Michael Freyberg, Felix Schmidt, Teng Liu, Sabine Osterhage, Mara Salvato, Andrea Merloni, Jeremy Sanders * Preprocessing / NRTA * Pipeline / ongoing processing * [[https://wiki.mpe.mpg.de/eRosita/PipeProc|processing wiki page]] * 947 * eRASS1 100 eROday DET chains * source catalogs * missing hardness ratios * mixed processing versions? * less cluster detections than in previous versions! * confirmed by Georg and Teng * SCUT/ECUT? * vs eFEDS [[attachment:ero100_eFEDS_20210921.pdf|ero100 vs eFEDS]] * announcing merged catalog to full team (erosita-members) * erbackmap in (almost) empty fields * hanging erbackmap runs (killed after >500minexecution time): sky tiles 139057, 037165 * run eFEDS CalPV in 947 pipeline * running Teng's pipeline on a shallower version of eFEDS * comparing 946/947 eRASS1 clusters in eFEDS field * vary task parameters in selected fields * check pipeline script * announcing merged catalog to full team (erosita-members) * erbackmap in (almost) empty fields * hanging erbackmap runs (killed after >500min execution time): sky tiles 139057, 037165 * pipeline issues / log file access * SOU chains / srctool * CalPV data * 946 * 001 (RXJ 1605) * TM5/7 are missing (caldb version mismatch) * eSASS tasks [[https://wiki.mpe.mpg.de/eRosita//eSASSdevel|eSASSdevel page]] / [[https://wiki.mpe.mpg.de/eRosita/pipelineErrorTracking|Error tracking page]] * prexec * now prevents 'hanging' START_... parameter files * param * how to handle concurrent log file accesses * finetuning wait time? * one pipeline log file per machine? * from previous meetings: * ermldet * Mara: RA/DEC/RADEC_ERR contained -999 (ermlet-1.47?) Julien will report details * Georg didn't fin -999 but just null values. Happens only very rarely. <- resolved * ftfindhotpix / caldb * Konrad's proposal (discarding/renaming flags) - should not break backward compatibility -> after 947 is out * evprep * Konrad provided scaled threshold map * dedicated meeting of Konrad and Ian * Susanne should put new threshold maps in * will only affect data after 47036 (is in eRASS4) - not need for eRASS:3 reprocessing * Update needed to address issue with bad pixel neighbours - could be implemented before eRASS3 947 reprocessing (see previous meeting) * License files (once again) * Correct name of license file * rename gpl-2.0.. -> COPYING? plus license.txt -> AUTHORS * Jeremy/Alain/HB/Johannes(?)/Bamberg(Sixte?) to discuss and make final decision * Jeremy will look how/whether the individual task authors are listed in SIXTE * * New user release * Test version by Ian * Reporting version number in user release (method by Philipp Weber) * recent task updates: * srctool-1.69 * prexec-1.10 * eroio-2.68 * ermldet-1.50 * erbackmap-1.21 (22?) * param-1.42, 143 (not yet in sw_archive) * HiPS maps * computing resources * Java / FTOOLS * Update frequency (already implemented) / rates images made by pipline? * exposure and rates maps are run twice per week * DAtool * server time-outs (causes?) * Backend * delivering 947 products * version numbers in file names * Frontend * emails * short term: change email text (HB will provide text) * delete button * full text link under buttons (data / delete) * medium term: send second (confirmation) email from backend * Request log: * links to region files * Deleting old (older than 3 months) request direcctorys * access privileges for www/DataRequests (Harald) * DataRequests dir should be group eroproc with no world read access * Script to obtain sky tiles (Miriam) * Longterm activities * '''[?]''' Moving task archive to git * evprep * '''[9]''' evprep: space invaders event flag * implemented - should be documented in next users release * more detailed, quantitative checks regarding improvement of data quality * evtool should update the gti according to this flag * wider issue: should discarded frames due to flag selection b tracked in GTI? * BTI/GTI: * '''[7]''' one PDF per eRASS in wiki * '''[7]''' PSF normalization * caldb vs. shaplet PSF (Antonis) * may be necessary to put 1 arcmin vs. 4 armin flux correction into shaplet caldb file (needs testing) -> longterm activity * ermldet * '''[3]''' Johan's proposal? Georg to discuss with Johan (long timescale project) * '''[7]''' Proposal by Florian: don't use fixed extrction radius but one dependent on flux * Time dependent RMFs/ARFs (srctool?) * '''[8]''' RMF * '''[7]''' ARF * '''[1]''' EV_WEIGHT: event based vignetting correction (would require dedicated tests) * '''[6]''' Background analysis: projecting closed data onto the sky (XMM SAS) * '''[6]''' ersensmap: implement pointing PSF * '''[3]''' ftfindhotpix: test and update algorithm * '''[1]''' Light leak in eSASS/pipeline? * '''[1]''' DAtool: filtering catalogs? * '''[6]''' add option to provide larger sky areas (Jeremy's script) ---- '''''Previous meeting:''''' * Q&A * Preprocessing / NRTA * Pipeline / ongoing processing * [[https://wiki.mpe.mpg.de/eRosita/PipeProc|processing wiki page]] * 947 * eRASS1 100 eROday DET chains * source catalogs * owner flag not set * ape tool data written to separate file * merged catalog * chain processing * hanging erbackmap in pathological fields (only exposure in corners) - ca. 10 sky tiles * hanging after apetool forced photometry runs - ca. 20 sky tiles * dividing srctool products into multiple files * Hermann: divide by eRASS (in SOU_947.PAR) <- agreed * Jeremy: tar all files of one source in srctool <- agreed * Alain: DAtsool backend needs to be adjusted * look at suggestion in 947 wiki page and make suggestion at data meeting * start eROday of eRASS1 * start on Dec. 11 * 946 * processed TM4 data * everything before threshod change is processed, will be continued when Ian is back * Cal pointings after orbit correction * RXJ 1605 (001) * TM5/7 errors * incompatible algos cal file? * will be resolved when Ian and Susanne re back * ART observations (946) * planning & PI file updated / running * disk space / copying * declination range 024 * group read access in archive? * eSASS tasks [[https://wiki.mpe.mpg.de/eRosita//eSASSdevel|eSASSdevel page]] / [[https://wiki.mpe.mpg.de/eRosita/pipelineErrorTracking|Error tracking page]] * from previous meetings: * ermldet * Mara: RA/DEC/RADEC_ERR contained -999 (ermlet-1.47?) Julien will report details * Georg didn't fin -999 but just null values. Happens only very rarely. <- resolved * ftfindhotpix / caldb * Konrad's proposal (discarding/renaming flags) - should not break backward compatibility -> after 947 is out * evprep * Konrad provided scaled threshold map * dedicated meeting of Konrad and Ian * Susanne should put new threshold maps in * will only affect data after 47036 (is in eRASS4) - not need for eRASS:3 reprocessing * Update needed to address issue with bad pixel neighbours - could be implemented before eRASS3 947 reprocessing (see previous meeting) * License files (once again) * Correct name of license file? * Jeremy/Alain/HB/Johannes(?)/Bamberg(Sixte?) to discuss and make final decision * Jeremy will look how/whether the individual task authors are listed in SIXT * New user release * Test version by Ian * Reporting version number in user release (method by Philipp Weber) * recent task updates: * evprep-3.44 * erolib-1.66 * catprep-2.8.3 * erbackmap-?? * user questions * [[https://lists.mpe.mpg.de/cgi-bin/mailman/private/erosita-helpdesk/|helpdesk]] * forum * Nico's question * can one subscribe to forums? * DAtool * Backend * delivering 947 products * version numbers in file names * speed improvements * Frontend * emails * short term: change email text * delete button * full text link under buttons (data / delete) * medium term: send second (confirmation) email from backend * Request log: * links to region files * access privileges for www/DataRequests (remind Harald) * Longterm activities * '''[?]''' Moving task archive to git * evprep * '''[9]''' evprep: space invaders event flag * implemented - should be documented in next users release * more detailed, quantitative checks regarding improvement of data quality * evtool should update the gti according to this flag * wider issue: should discarded frames due to flag selection b tracked in GTI? * BTI/GTI: * '''[7]''' one PDF per eRASS in wiki * '''[7]''' PSF normalization * caldb vs. shaplet PSF (Antonis) * may be necessary to put 1 arcmin vs. 4 armin flux correction into shaplet caldb file (needs testing) -> longterm activity * ermldet * '''[3]''' Johan's proposal? Georg to discuss with Johan (long timescale project) * '''[7]''' Proposal by Florian: don't use fixed extrction radius but one dependent on flux * Time dependent RMFs/ARFs (srctool?) * '''[8]''' RMF * '''[7]''' ARF * '''[1]''' EV_WEIGHT: event based vignetting correction (would require dedicated tests) * '''[6]''' Background analysis: projecting closed data onto the sky (XMM SAS) * '''[6]''' ersensmap: implement pointing PSF * '''[3]''' ftfindhotpix: test and update algorithm * '''[1]''' Light leak in eSASS/pipeline? * '''[1]''' DAtool: filtering catalogs? * '''[6]''' add option to provide larger sky areas (Jeremy's script)