Create signal quality test

[milktrader]

Suppose a long-only signal is in the market for 100 of 300 days.

The null hypothesis is that profit (or some other metric) from this signal are no better than chance.

I suppose the first pass should build a distribution from any random 100 values from 300, but maybe duration of consecutive days should be accounted for in drawing random samples (i.e., the signal was long for two periods, 80 days and 20 days, so the random draws should be subject to this constraint of consecutive days without replacement).

[milktrader]

There probably should be some naive (and impossible) assumptions about getting filled, etc with this test. The signal typically closes with the last print at which point it's not possible to get filled at that price. Essentially, the test assumes that you predict the signal with 100% accuracy and get filled at the closing price, regardless of whether that's even possible given the bid/ask nature of markets.

Modeling orders is a whole different enterprise than the goals of this test, which is a very rough first pass on whether a signal can identify a structure, or is no better than random.

[milktrader]

If returns are to track when signal is true then the special case when a signal is true on the last observation needs to be accounted for. Clearly, a next-day return cannot be attached to this signal since it's unknown. Given this, all signals (the one being tested and random ones) should assign the last observation to either

1. last signal to false
2. last return to 0

I'm inclined to the first option.