eSASS developers Zoom meeting
Start of meeting: 2020-11-10 13:30 (1:30pm)
Zoom link: https://zoom.us/j/7909307159?pwd=ZEdGeTJQUzdsRGpkd2Q5aUZXQmZZdz09
Participants: Ian Stewart, Hermann Brunner, Miriam Ramos, Ingo Kreykenbohm, Chandreyee Maitra, Andrea Merloni, Alain Gueguen, Mara Salvato, Jeremy Sanders, Michael Freyberg, Antonis Georgakakis
- Open actions
- Act20200929/1_all (Ian, Georg, Jeremy, Antonis, Miriam and Hermann will provide recomendations): make sure html task descriptions are current and consistent with user release
- first version of updates should be available before Nov 16
- update by Antonis
- task documention for eSASSuser + CalPV/001 should be sent be email, will be integrated in Web page by Miriam
- Miriam has initial template for updated Web page, to be discussed at meeting next week
- Act20201006/1_HB clean event file description, update source list description
- Act20201006/2_SF document cal files and bti
- static version on Web page / live version in wiki
- Act20201020/1_HB+SF check why some ObsIDs were only processed in 945
- GL: fixed in telgti-1.21 (not yet tested at MPE)
- Act20201102/1 AG+HB: test how many regions per request DATool can handle
- Act20200929/1_all (Ian, Georg, Jeremy, Antonis, Miriam and Hermann will provide recomendations): make sure html task descriptions are current and consistent with user release
- NRTA / preproc
- out of order events don't cause any trouble, speed issues solved
- Ongoing processing
- backup issues (discussion Harald/Achim/computer center)
- /ero_archive2-5 backups not current
- complaint by computer center about large number of files in DB
- excluding srtool products from backup? Not running SOU chain every day?
- moving products between disks while processing is running (DIRmove.csh)
- eRASS1
- 946 SOU chain processing complete (except for one sky tile)
- incorrectly set soft links to 945 srctool products (to be fixed by correction script)
- eRASS2
- Processing restarted after completion of eRASS1 SOU chain completion
- 946 srctool products
wrong srctool products names & links - correction script currently running
- sky tiles with incorrectly set soft links to 946 srctool products - correction scripts currently running
- Artefacts in eRASS2 Hips maps (Chandreyee):
- Strange feature in count images (232156 231153 231159)
- TM4 cal observation projected onto sky (eROday 45342) - fixed in tel-1.32
- funny exposure features (098099 and 096126)
- Ian's/Diogo's emails
- corrupted events in 237153, 231153, 232156, 231159, 223159, remove incorrect/obsolete slew staging files? - still under investigation
- TM5 stripes in SMC an across the sky (rerun with updated eRASS1 bti)
- TM4 timoff correction
- evprep path (v3.15), rerun affected erodays (45604-45606)
- New Wiki page?
- List an characterise types of data artefacts
- Track each instance a data artefact is discovered/corrected
- Start Wiki page(S) initially, investigate ticketing system
- Strange feature in count images (232156 231153 231159)
- eRASS1/2
- dalist should remove links to srctool products of previous run - not yet fixed
- RA/DEC_CEN not correctly set in 946 products
- 947 processing: start after eRASS1 946 srctool runs are complete
- adding additional eRASS1 BTI (will be finished on Thursday by Susanne)
- backup issues (discussion Harald/Achim/computer center)
- 001 processing:
- rerun to process missing ObsIDs (upgrade to telgti-1.21)
- Error in header keywords
- RA/DEC_CEN not set, occasionally wrong values - fix by correction script (Chandreyee)
- RXJ 1856 flux discrepancies
- Events with RA/DEC=0, evatt issue (update by Georg
- cause: out of sync values in gyro attitude - will be fixed in attprep
- other occurances: four cases after eROday 45000.
- analysis should be extended to all data, attprep should discard affected entries, and report about number of occurances
- other occurances: four cases after eROday 45000.
- cause: out of sync values in gyro attitude - will be fixed in attprep
- Events with RA/DEC=0, evatt issue (update by Georg
- flaregti in eFEDS field (Teng/Jeremy)
- parameters need to be adjusted to field size, should be in documentation, flaregti could also read the field size from input, Teng is using changed parameters for eFEDS
- Jeremy is investigating flaregti behaviour in survey ; parameters for 947
- simulating random light curves - ongoing
- testing whether flaregti improve detection
- eSASSusers testing
- Discrepancies in source list of demo script runs (different LAPACK versions)
- Removing build warnings in eSASS release (unused variables, etc.)
- cronjobs, scripts, etc.
- Running demoscript after nightly build? Ian will try this. - ongoing
- Script for evaluating evprep log info?
- task by Ian, Susanne's plots, Chandreyee's script?
eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page
- shapelib - Pavel's email
- catprep: set output and output2 (947 pipeline)
- telgti: attitude issues
- sky tile 062093 (events outside of sky tile)
- expmap: Werner's bug report
- detection task updates (ongoing)
- clobber option
- improved RADEC_ERR
- upper/lower count rate errors
- ermldet: ca. 2% of sources have positional error 0, some also have count error 0
- Recent task uploads
- tel-1.23 (Ian's update not yet included) - neds to be üut in 946 pipeline TODAY
- CalPV data release / public software release
- Meeting next week?
- CalPV reprocessing (config 001)
- some ObsIDs not processed in 946/001 + Frank's additions - reprocess with telgti-1.22
- Status of new users release
- srctool patch
- Inconsistent results with eSASSdevel and user release caldb?
- Solved by updating TM6 bad pixel caldb file
- Docker version
- created by Jonas, tested by Miriam and Alain, Wiki pages are ready, email will follow shortly
- srctool patch
- Updating documentation
- Task descriptions / default values
- Wiki documentation
- Public Web page
- DAtool
- Region requests issues
- owner filtering / evtool: use MPE owner eventflag
- Were all of Georg's bug reports fixed?
- catalogs can't yet be filtered, whole catalog will be returned
- Should this be implemented? So far only event files are filtered (via evtool).
- Catalog filtering could be implemented in Python based DAtool 2
- providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
- move simple copying request to Python / should everything be moved to Python? - ongoing
- DAtool / eROSE interface: requires change in fronted
- Frontend problem: first request by a user occasionally doesn't work
- download script (bash) should not be edite on Windowswindows
- bash script should not be opened with Windows editor!
- Region requests issues
- Camera resets / held cameras
reset |
TM |
timeoff |
badcamt |
report |
eDays released |
08/02 |
TM2 |
no |
yes |
yes |
|
08/08 |
TM4 |
no |
yes |
yes |
|
08/25 |
TM3 |
no |
yes |
- (no) |
|
08/27 |
TM4 |
no |
yes |
yes |
|
08/30 |
TM3 |
no |
yes |
yes |
|
09/05 |
TM5 |
no |
yes |
yes |
|
09/07 |
TM7 |
no |
yes |
yes |
|
09/08 |
TM1 |
yes |
yes |
yes |
|
09/08 |
TM4 |
yes |
yes |
yes 1) |
|
09/12 |
TM1 |
yes |
yes |
yes |
|
09/13 |
TM2 |
no |
yes |
yes |
|
10/03 |
TM6 |
no |
yes |
yes |
|
10/06 |
TM1/4/5/6/7 |
yes |
yes |
- |
yes (ITC reset) |
10/09 |
TM1 |
yes |
yes |
OAR-51 |
yes |
10/19 |
TM7 |
yes |
yes |
OAR-53 |
yes |
10/19 |
TM4 |
no |
yes |
OAR-52 |
yes (10/20: TM4 reset finished, 10/22: TM4 sync) |
10/21 |
TM6 |
yes |
yes |
OAR-54 |
yes |
11/03 |
TM2 |
no |
yes |
|
yes |
11/04 |
TM6 |
no |
yes |
|
yes |
11/06 |
TM7 |
no |
yes |
|
yes |
11/10 |
TM5 |
yes |
yes |
|
yes |
- Longterm activities
- ersensmap: implement pointing PSF
- expmap
- Bring coordinate transformations into line with evatt
- Parallel version
- evprep: experimental event flags
- tostprocessing: space invaders, frames with many counts, statistical event flagging
- Light leak mitigation
- Time dependent RMFs/ARFs (srctool?)
- Background analysis: projecting closed data onto the sky (XMM SAS)
- Moving to git
Previous meeting:
eSASS Q&A
- Open actions
- Act20200929/1_all (Ian, Georg, Jeremy, Antonis, Miriam and Hermann will provide recomendations): make sure html task descriptions are current and consistent with user release
- first version of updates should be available before Nov 16
- Act20201006/1_HB clean event file description, update source list description
- Act20201006/2_SF document cal files and bti
- static version on Web page / live version in wiki
- Act20201020/1_HB+SF check why some ObsIDs were only processed in 945
- GL: fixed in telgti-1.21 (not yet tested at MPE)
- Act20200929/1_all (Ian, Georg, Jeremy, Antonis, Miriam and Hermann will provide recomendations): make sure html task descriptions are current and consistent with user release
- New actions
- Act20201102/1 AG+HB: test how many regions per request DATool can handle
- lightcurves in autoreport are ok after
- Act20201102/1 AG+HB: test how many regions per request DATool can handle
- NRTA / preproc
- data in inspect directory / archiver update
- lightcurves in autoreport are ok after archiver update
- data in inspect directory / archiver update
- Ongoing processing
- backup issues (discussion Harald/Achim this week)
- /ero_archive2-5 backup may not be current
- moving products between disks while processing is running (DIRmove.csh)
- eRASS1
- 946 SOU chain processing of still missing sky tiles was started yesterday
- incorrectly set soft links to 945 srctool products - to be fixed by script (not yet started)
- eRASS2
- 946 SOU chains 99% complete, but:
wrong srctool products names & links - correction script currently running
- sky tiles with incorrectly set soft links to 946 srctool products - correction script currently running
- Artefacts in eRASS2 Hips maps (Chandreyee):
- Strange feature in count images (232156 231153 231159)
- Susanne will check whether bti need to be changed
- funny exposure features (098099 and 096126)
- check which cameras cause these features, related to light leak cameras? (Jeremy's GTI plot)
- corrupted events in 237153, 231153, 232156, 231159, 223159, remove incorrect/obsolete slew staging files? - still under investigation
- Strange feature in count images (232156 231153 231159)
- TM4 timoff correction
- evprep path (v3.15), rerun affected erodays
- Space invaders, frames with many counts, statistical flagging in postprocessing?, experimental flags in evprep?
- 946 SOU chains 99% complete, but:
- eRASS1/2
- dalist should remove links to srctool products of previous run - not yet fixed
- RA/DEC_CEN not correctly set in 946 products
- 947 processing: start after eRASS1 946 srctool runs are complete
- create new gyro boresight version with validity start at begin of eRASS1, the run a few scans and test the boresight before proceeding
- adding additional eRASS1 BTI
- backup issues (discussion Harald/Achim this week)
- 001 processing:
- run on missing ObsIDs (upgrade to telgti-1.21)
- rerun TM5 eFEDS (bti)
- Error in header keywords
- RA/DEC_CEN not set, occasionally wrong values - fix by correction script
- RXJ 1856 flux discrepancies
- Events with RA/DEC=0, evatt issue (Georg's input needed), updating GTI in case of bad attitude?
- corrupted values in raw and intermediate attitude, will be fixed in attprep
- testing how often this happened: plot merged attitude file (Georg will have a look)
- corrupted values in raw and intermediate attitude, will be fixed in attprep
- Events with RA/DEC=0, evatt issue (Georg's input needed), updating GTI in case of bad attitude?
- flaregti in eFEDS field (Teng/Jeremy)
- parameters need to be adjusted to field size, should be in documentation, flaregti could also read the field size from input, Teng is using changed parameters for eFEDS
- Jeremy is investigating flaregti behaviour in survey ; parameters for 947
- simulating random light curves
- testing whether flaregti improve detection
- cronjobs, scripts, etc.
- Running demoscript after nightly build? Ian will try this.
- Script for evaluating evprep log info?
- plot script on output of Chandreyee's script?
eSASS tasks (ongoing/planned) / current processing errors - eSASSdevel page / Error tracking page
- shapelib - Pavel's email
- catprep: set output and output2 (947 pipeline)
- telgti/evtool: attitude issues
- detection task updates (ongoing)
- clobber option
- improved RADEC_ERR
- upper/lower count rate errors
- ermldet: ca. 2% of sources have positional error 0, some also have count error 0
- Recent task uploads
- CalPV data release / public software release
- CalPV reprocessing (config 001)
- some ObsIDs not processed in 946/001 - reprocess with telgti-1.21
- List of ObsIDs to be released (additions by Frank)
- Status of new users release
- srctool patch
- Inconsistent results with eSASSdevel and user release caldb?
problem: version.for has same name in each task, should be renamed to <task>_version.for
- Docker version
- created by Jonas, tested by Miriam and Alain, Wiki pages are ready, email will follow shortly
- srctool patch
- Status of calibration
- Static CalPV release caldb at /home/erosita/sw/caldb (renaming it?)
- config script allows to specify path to caldb (docu in wiki)
- Updating documentation
- Task descriptions / default values
- Wiki documentation
- Public Web page
- CalPV reprocessing (config 001)
- DAtool
- Region requests issues
- owner filtering / evtool: use MPE owner eventflag
- Were all of Georg's bug reports fixed?
- catalogs can't yet be filtered, whole catalog will be returned
- Should this be implemented? So far only event files are filtered (via evtool).
- Catalog filtering could be implemented in Python based DAtool 2
- providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
- move simple copying request to Python / should everything be moved to Python? - ongoing
- DAtool / eROSE interface: requires change in fronted
- Frontend problem: first request by a user occasionally doesn't work
- download script (bash) should not be edite on Windowswindows
- bash script should not be opened with Windows editor!
- Region requests issues
- Camera resets / held cameras
reset |
TM |
timeoff |
badcamt |
report |
eDays released |
08/02 |
TM2 |
no |
yes |
yes |
|
08/08 |
TM4 |
no |
yes |
yes |
|
08/25 |
TM3 |
no |
yes |
- (no) |
|
08/27 |
TM4 |
no |
yes |
yes |
|
08/30 |
TM3 |
no |
yes |
yes |
|
09/05 |
TM5 |
no |
yes |
yes |
|
09/07 |
TM7 |
no |
yes |
yes |
|
09/08 |
TM1 |
yes |
yes |
yes |
|
09/08 |
TM4 |
yes |
yes |
yes 1) |
|
09/12 |
TM1 |
yes |
yes |
yes |
|
09/13 |
TM2 |
no |
yes |
yes |
|
10/03 |
TM6 |
no |
yes |
yes |
|
10/06 |
TM1/4/5/6/7 |
yes |
yes |
- |
yes (ITC reset) |
10/09 |
TM1 |
yes |
yes |
OAR-51 |
yes |
10/19 |
TM7 |
yes |
yes |
OAR-53 |
yes |
10/19 |
TM4 |
no |
yes |
OAR-52 |
yes (10/20: TM4 reset finished, 10/22: TM4 sync) |
10/21 |
TM6 |
yes |
yes |
OAR-54 |
yes |
11/03 |
TM2 |
no |
yes |
|
yes |
11/04 |
TM6 |
no |
yes |
|
yes |
11/06 |
RM7 |
no |
- |
|
no |
- Longterm activities
- ersensmap: implement pointing PSF
- expmap
- Bring coordinate transformations into line with evatt
- Parallel version
- Light leak mitigation
- Time dependent RMFs/ARFs (srctool?)
- Background analysis: projecting closed data onto the sky (XMM SAS)
- Moving to git