#acl EroCatGroup:read,write,revert All:read == Weekly eSASS developers meeting == '''Start of meeting: 2021-02-16 12:30 <<<< meeting starts earlier this time and will end before 14:00''' Zoom link: https://zoom.us/j/97439409844?pwd=YXVkK0hBMnpybm5tL2VyYTh2Qld2UT09 Participants: Ian Stewart, Hermann Brunner, Miriam Ramos, Alain Gueguen, Konrad Dennerl, Mara Salvato, Sabine Osterhage, Susanne Friedrich, Jeremy Sanders. Chandeyee Maitra, Andrea Merloni, Michael Freyberg * Open actions from previous developers meetings * Act20200929/1_all make sure html task descriptions are current and consistent with user release * updates by Jeremy and Ian (evtool, expmap, catprep) * still missing: Hermann (radec2xy), Georg (detection + evatt) * Act20201006/1_SF clean event file description, update source list description * Preprocessing / NRTA * eSASS tasks (ongoing/planned) / current processing errors - [[https://wiki.mpe.mpg.de/eRosita/EroCat/eSASSdevel|eSASSdevel page]] / [[https://wiki.mpe.mpg.de/eRosita/pipelineErrorTracking|Error tracking page]] * eRASS1/2 947 code updates/code freeze * perhaps small evprep updates * expmap: * parallel version (7 cores) * withinputmaps=yes * testing? * catprep: making number of bands variable (1..5) - in progress * detection task updates (ongoing, perhaps ready for 947) * positional error / count error 0 * upper/lower count rate errors - will also be propagated by catprep * -> ermldet new task version * ftfindhotpix / badpix / caldb access in general * accessing highest validity date file vs all overlapping file * TMIN/TMAX columns not use in current version * Ian will make ero_cifsl return files in decending validity start date order * Temperature dependent energy calibration * Konrad will provide update of task energy (will require new caldb file) * Pattern update: read badpix start and eion) * fftw version (Alain) * different ermldet results on old/new machinnd times * srctool vs. apetool exposure (help desk questes may be related to fftw version (results with v3.3.9 are ok) * v3.3.8 in current user release should be upgraded - should be documented (eSASSnews) * Docker version needs to be updated * Check version in pipeline eSASS * eSASSuser on Ubuntu 20 - after fftw issue resolved * eropipe, proc, prexec - Alain * telstage, dalist, rmvers - Miriam * test environment, current task versions -> eSASSdevel/task archive * Recent task updates: * [[https://lists.mpe.mpg.de/cgi-bin/mailman/private/erosita-sass/2021-February/001151.html|erolib-2.40, evtool-2.19, evprep-3.19]] * Pipeline / ongoing processing * [[https://wiki.mpe.mpg.de/eRosita/PipeProc|processing wiki page]] * performance issues? * replacement for get_dir routine? * 947 test runs * eRASS1/2, eRASS:2 (first 100 eDays) are in archive * HiPS (eRASS1/2), DAtool, ProcStat (ok) * DAtool (947): ImageTot -> Image, energy bands: 0-9, a-d for e?01..., e?02..., s?02... * HiPS energy bands? stay with the same bands for now (more bands perhaps later) * Astrometry? * eRASS2/3 946 exposure deficit * use latest HK file versions + tel-1.24 (telgti border issues) * fixed in 947 / no full 946 rerun of affected data planned * Problem might still exist in recently processed data? e.g., eROday 46283, HB will have look * tel-1.24 not yet updated * Missing soft sources (Frank) * pre-946 processing versions contained CalPV test scans * confirmation: rerun test scans in 946 pipeline (or 001?) * Survey scans on Dec 8 and Dec 10 should not be part of eRASS1 * em00 files end of Nov. cover part of LMC. These eDays should be rerun in 001 pipeline * TM7 bad pixels * caldb was updated, eDays 46199+ restarted * data look clean now * 949 TEL chain * for Konrad's temperature dependent energy calibraton * run TEL chain up until pattern * CalPV + recent data * Data access * eRASS event files / images * FWC data * EDR * Web pages, documentation * Validation of CalPV/c001 data: guidlines by eSASS team * DAtool / ProcSTat * access restrictions for data request logs + project descriptions (discussion ongoing - details tbd) * providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine * move simple copying request to Python / should everything be moved to Python? - ongoing * DAtool / eROSE : requires change in frontend * Longterm activities * '''[?]''' Moving task archive to git * evprep * '''[9]''' evprep: space invaders event flag * evtool should update the gti according to this flag * post-processing: frames with many counts? * BTI/GTI: * PDF for CalPV linked on wiki page (done?) * '''[7]''' one PDF per eRASS in wiki * ermldet * '''[8]''' ca. 2% of sources have positional error 0, some also have count error 0 * can't be fixed quickly, may need replacement of algorithm * Johan's proposal? Georg to discuss with Johan (long timescale project) * Time dependent RMFs/ARFs (srctool?) * '''[8]''' RMF * '''[7]''' ARF * '''[6]''' Background analysis: projecting closed data onto the sky (XMM SAS) * '''[4]''' ersensmap: implement pointing PSF * '''[3]''' ftfindhotpix: test and update algorithm * '''[1]''' Light leak in eSASS/pipeline? * '''[1]''' DAtool: filtering catalogs (separate from DAtool)? ---- '''''Previous meeting:''''' * Open actions from previous developers meetings * Act20200929/1_all make sure html task descriptions are current and consistent with user release * updates by Jeremy and Ian (evtool, expmap, catprep) * still missing: Hermann (radec2xy), Georg (detection + evatt) * Act20201006/1_SF clean event file description, u pdate source list description * Preprocessing / NRTA * Noise/offset maps? * eSASS tasks (ongoing/planned) / current processing errors - [[https://wiki.mpe.mpg.de/eRosita/EroCat/eSASSdevel|eSASSdevel page]] / [[https://wiki.mpe.mpg.de/eRosita/pipelineErrorTracking|Error tracking page]] * eRASS1/947 code updates/code freeze (Ian's email) * perhaps small evprep updates * expmap: * parallel version (7 cores) * second order parallelization (by GTI) * withinputmaps=yes * testing? * catprep: making number of bands variable (1..5) - in progress * detection task updates (ongoing, perhaps ready for 947) * clobber option * improved RADEC_ERR, upper/lower count rate errors * Pipeline tasks (shadow developer/pipeline operator) * eropipe, proc, prexec - Alain * telstage, dalist, rmvers - Miriam * test environment, current task versions -> eSASSdevel/task archive * Recent task updates: * expmap: v2.1.1-3, v2.2, v1.33.4 * Pipeline / ongoing processing * [[https://wiki.mpe.mpg.de/eRosita/PipeProc|processing wiki page]] * performance issues? * replacement for get_dir routine? * ds53 speed issues? * 947 test runs * eRASS1/2, eRASS:2 (first 100 eDays) are in archive * HiPS (eRASS1/2), DAtool, ProcStat (ok) * DAtool (947): ImageTot -> Image, energy bands: 0-9, a-d for e?01..., e?02..., s?02... * HiPS energy bands? * Astrometry? * eRASS3 (946) exposure deficit * caused (in part) by incomplete/not-current HK file versions - fixed * telgti issues at border of planning files (fixed in tel-1.24) * 46008 and 46085 rerun in eRASS3 946 processing - HiPS maps? * similar exposure deficits in eRASS1/2 * Missing soft sources (Frank) * Ian, Chandreyee, Frank investigated * probable cause: previous processings contained CalPV test scans * confirmation: rerun test scans in 946 pipeline (or 001?) * bad pixels * TM7: * 42 bad pixels in caldb (onboard vs. caldb?) * onboard: 40 bad pixels * How to proceed with eDay 46199? * Susanne will set vsd to beginning of eROday 46199 * Temperature dependence in task energy (HK access) * should go in 947 pipeline * Konrad proved list of eROdays to Hermann - TEL_949 chains currently running * Zip pattern analysis * Susnanne + Chandreyee will discuss * Discrepancies in source list of demo script runs / ermldet uninitialized variables * Still different results of same binaries on different machines * Different from pervious problem, all sources are affected (parameter are sightly different) - high priority * Confirm whether ermldet patch changes the eFEDS source list (Teng?) * evprep summary reports * will be updated by Chandreyee to check TM5/7 against successive eROdays * EDR * Web pages, documentation * Validation of CalPV/c001 data: guidlines by eSASS team * https://wiki.mpe.mpg.de/eRosita/guide_validationEDR * DAtool * access restrictions sto data request logs + project descriptions (discussen ongoing- details tbd) * providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine * move simple copying request to Python / should everything be moved to Python? - ongoing * DAtool / eROSE interface: requires change in frontend * Longterm activities * '''[?]''' Moving task archive to git * evprep * '''[9]''' evprep: space invaders event flag * which flag location: bit 3, parameter in pipeline needs to be turned on * evtool should update the gti according to this flag * post-processing: frames with many counts? * BTI/GTI: * PDF for CalPV linked on wiki page (done?) * '''[7]''' one PDF per eRASS in wiki * ermldet * '''[8]''' ca. 2% of sources have positional error 0, some also have count error 0 * can't be fixed quickly, may need replacement of algorithm * Johan's proposal? Georg to discuss with Johan (long timescale project) * Time dependent RMFs/ARFs (srctool?) * '''[8]''' RMF * '''[7]''' ARF * '''[6]''' Background analysis: projecting closed data onto the sky (XMM SAS) -> 6 * '''[4]''' ersensmap: implement pointing PSF * ftfindhotpix: test and update algorithm * '''[1]''' Light leak in eSASS/pipeline? * '''[1]''' DAtool: filtering catalogs (separate from DAtool)? -> 1