Enable serial CTE correction on WFC subarrays

[jryon]

We did not enable serial CTE correction on WFC subarrays. One reason was Michele found that rotating non-square arrays was hard to implement in C: "It turns out that rotating rectangular matrices (M x N where M != N) is complex to do and involves a trade-off between memory and speed.  I have been researching this, and it looks like I should use some library function (e.g., linear algebra package) to do the work.  Square matrices are sooooo easy in comparison." In a later email she mentions having added some code to do the rotation that she would then remove, but I don't know if it was complete, or committed at any point.

Another reason had to do with expectations for what CALACS should do with subarrays, given that the keyword rules currently set PCTECORR = OMIT for subarrays. The reason for that is because they cannot have bias-striping correction run on them in the way CALACS does for full-frames. That is why acs_destripe_plus exists. However, when PCTECORR is set to PERFORM for a supported subarray, then parallel CTE correction can be run on it by acs_destripe_plus.

So, I think if we can figure out how to rotate subarrays to perform the serial CTE correction within CALACS, then we should extend it to all supported subarrays.

We also added a check to skip serial CTE correction for all subarrays. The discussion on that in the old issue starts here: #577 (comment)

xref

• HLA-1133/1241/1288 Implementation of ACS Serial CTE correction #609

[pllim]

On hindsight, we should not have coded so many checks into the CTE step (or any step). That is what the switches are for. If it is set to PERFORM by DMS, we perform it. If people set it to PERFORM without knowing better, that is their problem. To enable CTE correction for subarrays, we will need to clean this up.

Also, aren't most subarrays square anyway? 512 and 1k are squares. Only 2k is the problem? Michele is right that rectangles are added complications, but making an extra temporary 2-D float array nowadays isn't a hardware problem anymore for ACS (cough, Roman, cough), so I won't worry too much about that.

[pllim]

So the meat of the rotation are here:

hstcal/pkg/acs/lib/acscte/dopcte-gen3.c

Line 406 in 42c363a

static int rotateAmpData(FloatTwoDArray * amp, const unsigned ampID)

hstcal/pkg/acs/lib/acscte/dopcte-gen3.c

Line 434 in 42c363a

static int derotateAmpData(FloatTwoDArray * amp, const unsigned ampID)

I will have to kill some brain cells to figure out why these cannot be applied to subarrays. They look pretty generic but I also haven't thought too much about it. Given the very specific nature of needs here, if we have to write new code for the rectangular subarray, we can probably brute force it anyway instead of writing a generic algorithm.

Definitely some time need to be spent on this.

So I think at a high level, the summary of work needed here:

1. Remove subarray checks in CTE code so entire algorithm is solely controlled by PCTECORR. (EDIT: Do this after Step 4)
2. Compile a list of subarrays that would go through CTE correction and their respective dimensions.
3. Investigate the work needed to make rotation functions work for all subarrays.
   a. Enable serial CTE correction on WFC subarrays #700 (comment)
4. Do the work identified in (3).
5. Collect test cases from INS to run new code on.
6. Work with INS to fine-tune subarray serial correction; do we need new PCTETAB?
7. Add new test cases to ACS/WFC regression test suite and update existing truths if needed.
8. Make a release.
9. Update data handbook?
10. ???
11. Profit!!!

[jryon]

The currently supported 512 and 1K subarrays are not square. They're 2048 x 512 and 2048 x 1024 (without prescans). The 2K are squares:
https://hst-docs.stsci.edu/acsihb/chapter-7-observing-techniques/7-7-acs-apertures#id-7.7ACSApertures-table7.7

This list sounds good. I'd suggest leaving the "Remove subarray checks" till the end, or at least keeping it around, commented out or whatever, until we have done 2-4. I think the reason for at least some of the subarray checks is so the parallel correction can still be run on supported subarrays with acs_destripe_plus. But I may be wrong.

I can work on 2.

We shouldn't need a new PCTETAB. The timing between supported subarrays and full-frames is identical, so CTE behaves the same in both, there are just fewer rows read out.

[jryon]

Here's the list for item 2 of the list above:

WFC1-POL0UV 2048 x 2048
WFC1-POL0V 2048 x 2048
WFC2-POL0UV 2048 x 2048
WFC2-POL0V 2048 x 2048
WFC1-POL60UV 2048 x 2048
WFC1-POL60V 2048 x 2048
WFC2-POL60UV 2048 x 2048
WFC2-POL60V 2048 x 2048
WFC1-POL120UV 2048 x 2048
WFC1-POL120V 2048 x 2048
WFC2-POL120UV 2048 x 2048
WFC2-POL120V 2048 x 2048
WFC1A-2K 2048 x 2048
WFC1B-2K 2048 x 2048
WFC1A-1K 2048 x 1024
WFC1B-1K 2048 x 1024
WFC1A-512 2048 x 512
WFC1B-512 2048 x 512
WFC2C-2K 2048 x 2048
WFC2D-2K 2048 x 2048
WFC2C-1K 2048 x 1024
WFC2D-1K 2048 x 1024
WFC2C-512 2048 x 512
WFC2D-512 2048 x 512
WFC1-2K 2048 x 2046
WFC2-2K 2048 x 2046
WFC1-IRAMPQ 2048 x 2046
WFC1-MRAMPQ 2048 x 2046
WFC2-MRAMPQ 2048 x 2046
WFC2-ORAMPQ 2048 x 2046

[pllim]

Natural orientation

Ref: https://hst-docs.stsci.edu/acsihb/chapter-4-detector-performance/4-2-the-ccds#id-4.2TheCCDs-fig4.2

Flipped for parallel (Y) CTE correction

Flipped for serial (X) CTE correction

Initial battle plan

• Move the rotation and flipping functions out of the bowels of CTE code into its own module in https://github.com/spacetelescope/hstcal/tree/main/lib (and accompanying header file)
• Write unit tests for them like https://github.com/spacetelescope/hstcal/blob/main/pkg/imphttab/test_getphttab.c
• See if existing code naturally work with rectangle already (given total buffer size is the same in before/after, they might but we need to test and see)
• If it does not, modify the code to account for rectangular dimensions (skip this if it does already)
• Modify CTE code to use these functions. I don't see the point of moving the pixels around twice for serial CTE, so I want to refactor so that serial only do it once to get the array into the correct orientation, which means also have to make sure they get un-oriented correctly after only one array manipulation step
• Stretch goal: See if we can integrate the C unit tests into CI (see TST: Enable C testing #704)