eSASSdevel_210525 - EROSITAwiki

Weekly eSASS developers meeting

Start of meeting: 2021-05-25 13:30

Zoom link: https://zoom.us/j/97439409844?pwd=YXVkK0hBMnpybm5tL2VyYTh2Qld2UT09

Participants: Ian Stewart, Hermann Brunner, Miriam Ramos, Alain Gueguen, Konrad Dennerl, Sabine Osterhage, Susanne Friedrich, Chandeyee Maitra, Michael Freyberg, Jeremy Sanders, Hatald Baumgartner, Andrea Merloni, Felix Schmidt, Teng Liu, Mara Salvato

Preprocessing / NRTA
eSASS tasks (ongoing/) / current processing errors - eSASSdevel page / Error tracking page
- Missing Task updates for 947 pipeline?
  - TEL chain:
    - evprep (see below)
    - Pattern: caldb was updated - Konrad will provide new version
    - ftfindhotpix-1.12 make sure there are no issues
  - EXP chain:
    - make sure header keywords are correct (HB, GL, CM)
  - DET chain:
    - expmap set parameter to read badpix from extension - option removed in latest version
    - ermldet: default libs changed for fftw in eSASSdevel (move to 947 pipeline eSASS)
      - copy default.libs from eSASSdevel to eROproc (check whether fftw is from external or system, external referred)
    - catprep (extra columns)
    - evtool: DATE-OBS, DATE-END keywords / expmerge??
      - Ian will write TSTOP=TSTART in case of empty GTI
    - Create/update 947 chain script
  - SOU chain
    - lightcurve script by Johannes?
      - one bin for each pass o a source through the fov (Jeremy looks at srctool code)
      - Max Lik. techique from paper by Axel (seem to work in exerimental code, courld be put in production srctool)
    - subdirectories for srctool products:
      - split by eRASS (and by groups of each 100 sources)
    - specfit?
EDR discussion with Harald
eSASS continued
- evprep
  - thresholding (running 946+947 in parallel, Ian's summary) - use corralgo algorithm (TM5/7 special treatment)
  - SPLT2NOI HK value: parameter for evprep to pass location of file to keep track last good value of threshold scaling - should be in
  - GTI for RMIDLE/other 'funny' modes?
  - Threshold columns? (not for 947) How to handle in expmap & srctool? Offline discussion...
- erotools (Ian): setbadpix, flag_report, date2sec (<- time_mod in task general includes leap seconds)
- ermldet-1.43: asymmetric errors, etc. - is now in eSASSdevel
- apetool, shapelib: compatible verson is now in eSASSdevel
- catprep: update to accommodate new ermldet columns
  - should copy all columns provided by ermldet-1.43
- expmerge: fix OWNER keyword (HB)
- Lightcurve Python script by Johannes (for 947 processing)
- Changed BADPIX format: ftfindhotpix-1.11 and srctool-1.65 moved to 946 pipeline
- New pattern version for 947 (ccf update)
- Energy: accessing correct offset map?
  - General improvement, not for 947
  - Energy should have a task parameter which points to Ingo's table of offset map versions
- ftfindhotpix: remove hardcoded version number
- Pipeline testing (Alain...)
  - telselect crash (which task versions?)
- Recent task updates:
  - erolib-2.53
  - onetools-1.9
  - evprep-3.25
  - eroio-2.56
- Share pipeline par files with IKI (Pavel)
- Michaels statistical GTI analysis: Michael will look at two eROdays in detail and try to identify specific errors (wrong exposure)
- Pipeline / ongoing processing
  - processing wiki page
  - 946
    - held cameras
      - eROdays 46799-46845 reprocessed (Chandreyee)
    - caldb: bright pixels/columns, Zip patterns
    - processing errors, backlogs, etc.
      - TEL_946 chain upgrade to ftfindhotpix to v1.11 caused TEL chain crashes (fixed/reprocessed)
    - disk copying (Harald)
  - 947
    - test runs / time scale / code freeze / reporting on test run results
    - HiPS maps header keywords
  - 001 (CalPV) reprocessing
    - EXP chains hanging?
  - Orbit correction/cal observations
    - 001 processing for RXJ 2143 (Frank)
    - Puppis A, A 3266
EDR
- EDR Web pages, page counters (https://de.wikipedia.org/wiki/Matomo), etc.
  - GL talked to Adriana, REAMDE files still needed
  - also copy time synchronization files
- Orbit files - just Adriana's files, place on EDR Web page
  - Offline discussion...
- Forum software
- Data issues
  - CalPV TSTART/TSTOP/DATE-OBS,DATE-END and times, header keywords
    - Fixing/documenting wrong time keywords
    - Investigate whether the issue persists in current eSASS versions
    - OWNER keyword: fixed in EDR data (Chandreyee) / correct expmerge task
    - GTI when in PMIDLE? should be removed in evprep, how to handle other 'strange' modes? -> 947?
      - Document for EDR (Ian's email)
    - New November planning file
      - eDAYs 43637-43654, new ObsIDs: 700199, 700200 (replaces 700195), also affected: 700193, 700194 TEL/EXP chain rerun, results ok?
    - TeX reports / Susanne's Python script
DAtool / ProcStat
- User emails (response time)? DATOOL emails / help-desk
- access restrictions for data request logs + project descriptions
- providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
- move simple copying request to Python - ongoing
Longterm activities
- [?] Moving task archive to git
- evprep
  - [9] evprep: space invaders event flag
    - evtool should update the gti according to this flag
    - post-processing: frames with many counts?
  - BTI/GTI:
    - PDF for CalPV linked on wiki page (done?)
    - [7] one PDF per eRASS in wiki
- ermldet
  - [8] ca. 2% of sources have positional error 0, some also have count error 0
    - can't be fixed quickly, may need replacement of algorithm
    - Johan's proposal? Georg to discuss with Johan (long timescale project)
- Time dependent RMFs/ARFs (srctool?)
  - [8] RMF
  - [7] ARF
- [7] EV_WEIGHT: event based vignetting correction
- [6] Background analysis: projecting closed data onto the sky (XMM SAS)
- [4] ersensmap: implement pointing PSF
- [3] ftfindhotpix: test and update algorithm
- [1] Light leak in eSASS/pipeline?
- [1] DAtool: filtering catalogs (separate from DAtool)?

Previous meeting:

Old actions:
- EDR task descriptions etc.
Preprocessing / NRTA
eSASS tasks (ongoing/) / current processing errors - eSASSdevel page / Error tracking page
- Missing Task updates for 947 pipeline?
  - ermldet build on (ds53): fftw3 from external; default.libs in eSASSdev modfied / eROproc uses system versions?
  - evprep (see below)
  - catprep (extras columns)
  - lightcurve script by Johannes?
  - pattern
- evprep
  - thresholding (running 946+947 in parallel, Ian's summary) - use corralgo algorithm (TM5/7 special treatment)
  - SPLT2NOI HK value: parameter for evprep to pass location of file to keep track last good value of threshold scaling - should be in
  - Threshold columns? (not for 947) How to handle in expmap & srctool? Offline discussion...
- erotools (Ian): setbadpix, flag_report, date2sec (<- time_mod in task general includes leap seconds)
- ermldet: asymmetric errors, etc. - in task archive but not yet in eSASSdevel
- apetool, shapelib: in task archive but no yet in eSASSdevel (wrong dir/file name)
- catprep: update to accommodate new ermldet columns
  - should copy all culmns provided by ermldet-1.43
- Lightcurve Python script by Johannes (for 947 processing)
- Changed BADPIX format
  - move ftfindhotpix-1.11 and srctool-1.65 to 946 pipeline (done)
    - set expmap to read badpix extension
- New pattern version for 947 (ccf update)
- Energy: accessing correct offset map?
  - General improvement, not for 947
  - Energy should have a task parameter which points to Ingo's table of offset map versions
- Pipeline testing (Alain...)
  - telselect crash (which task versions?)
- Recent task updates:
  - ermldet-1.42, eroio-2.54, apetool-1.12, shapelib-2.41 (asymmetric errors, not yet in eSASSdevel)
  - flaregti-1.23 (expmap_io -> expmap_legacy)
- Share pipeline par files with IKI (Pavel)
- Michaels statistical GTI analysis: Michael will look at two eROdays in detail and try to identify specific errors (wrong exposure)
- Pipeline / ongoing processing
  - processing wiki page
  - 946
    - held cameras
    - caldb: bright pixels/columns, Zip patterns
    - processing errors, backlogs, etc.
    - Hips map: 021 band count excess - with new evprep will be fixed in 947
    - Exposure map discontinuity: resolved? Should not happen in 947 (Chandreyee)
    - disk copying (Harald)
  - 947
    - test runs / time scale / code freeze / reporting on test run results
    - HiPS maps header keywords
  - 001 (CalPV) reprocessing
  - Orbit correction/cal observations
    - 001 processing for RXJ 2143 (Frank)
    - Puppis A, A 3266 (planning file?)
EDR
- EDR Web pages, page counters (https://de.wikipedia.org/wiki/Matomo), etc.
- Orbit files - just Adriana's files, place on EDR Web page
  - Offline discussion...
- Forum software
- Data issues
  - CalPV TSTART/TSTOP and times, header keywords, Puppis A
    - Fixing or documenting wrong time keywords (either globally and/or in the TeX reports)?
    - Investigate whether the issue persists in current eSASS versions
    - GTI when in PMIDLE? should be removed in evprep, how to handle other 'strange' modes? -> 947?
      - Which EDR data are affected? How to document?
    - New November planning file
      - 001 pipeline needs to be rerun for eDAYs 43637-43654, new ObsIDs: 700199, 700200 (replaces 700195), also affected: 700193, 700194 -> HB, Chandreyee needs to rerun header correction script
  - TeX reports / Susanne's Python script
DAtool / ProcStat
- User emails?
- access restrictions for data request logs + project descriptions
- providing large multi-sky tile images - Jeremy will share his Python script with Alain and Sabine
- move simple copying request to Python - ongoing
Longterm activities
- [?] Moving task archive to git
- evprep
  - [9] evprep: space invaders event flag
    - evtool should update the gti according to this flag
    - post-processing: frames with many counts?
  - BTI/GTI:
    - PDF for CalPV linked on wiki page (done?)
    - [7] one PDF per eRASS in wiki
- ermldet
  - [8] ca. 2% of sources have positional error 0, some also have count error 0
    - can't be fixed quickly, may need replacement of algorithm
    - Johan's proposal? Georg to discuss with Johan (long timescale project)
- Time dependent RMFs/ARFs (srctool?)
  - [8] RMF
  - [7] ARF
- [7] EV_WEIGHT: event based vignetting correction
- [6] Background analysis: projecting closed data onto the sky (XMM SAS)
- [4] ersensmap: implement pointing PSF
- [3] ftfindhotpix: test and update algorithm
- [1] Light leak in eSASS/pipeline?
- [1] DAtool: filtering catalogs (separate from DAtool)?

EROSITAwiki: eSASSdevel_210525 (last edited 2021-05-25 13:06:11 by HermannBrunner)