Drop NAs in the ESTIMATE() function

[fuleky]

Dear Andrea,

Thank you for your work on the bimets package. I am not sure if you accept feature requests, but the one I outline below would be really handy. Thank you for considering it. If there is an existing workaround, please let me know.

Feature request: Drop NAs in the ESTIMATE() function

Background:
The ESTIMATE function expects data that has no NAs in TSRANGE. In practice, variables have different start and end dates. GETRANGE can get the intersection of dates. But a typical model contains various lags of variables and GETRANGE does not consider lags in the model.

Problem:
ESTIMATE throws an error when it encounters NAs.
TSRANGE can be set globally in ESTIMATE function, but this risks collinearity.

Example:
Say there are three equations:

1. first eq: the data for the first has a shorter range, e.g. 2000-2020.
2. second eq: the data for the second has the same range as the first, but the equation contains lags of variables, e.g. TSLAG(x, 4).
3. third eq: the third one contains a (dummy) indicator variable with 1 before the data range of the first two equation, i.e. ones before 2000, and zeros after.
   Setting global TSRANGE to overlap with the range of equation 1 will result a variable with only zeros in equation 3 (apparent collinearity). And equation 2 will throw an error because the lags are not accommodated by TSRANGE.
   Setting global TSRANGE to overlap with the range of equation 3 will result in missing values in equation 1, since the range would start before 2000.

Proposed solution:
These problems could be solved by having the following options/arguments in the ESTIMATE function:

1. A logical argument to drop data rows (time periods) containing NAs from each equation. For this to accommodate lags, the full model matrix with all the lags should be first assembled before rows with NAs are dropped. If this is implemented at the equation level, that would still allow TSRANGE to vary from equation to equation. (Internal NAs could still throw an error).
2. A logical argument to drop data columns (variables) that cause collinearity when setting TSRANGE globally (just like stats::lm() does).

The two could also be combined to just drop all NAs in the model matrix before estimation. Or this could also just be a separate function that can be applied to a model with data. The function could print messages listing equations whose data and TSRANGE was altered.

[andrea-luciani]

Dear Prof. Fuleky.

Thank you again for your renewed interest in the bimets package.

I've got your point. We already received a similar request from other users to add an automatic NAs removal feature in the bimets ESTIMATE function. Due to the large number of ESTIMATE options (e.g., coefficient restrictions, IV, error auto-correlation, PDL, etc.), it is not an easy task to code an automatic removal of observations that produce NAs in regressors' assessment, as it requires a careful modification in the underlying core code. Anyhow, we will try to insert that feature in the upcoming bimets versions. As I said, we already had an initial brainstorming on how to implement this feature, without breaking the code, due to previous requests from other users.

In the meantime, you can take a look at the MDL capability that allows users to specify a different TSRANGE for different equations, as you can see on page 59 of the Reference Manual (https://cran.r-project.org/web/packages/bimets/bimets.pdf).

Back to your example, after moving the global TSRANGE definition from the ESTIMATE function call (pag. 55 in the ref. manual) to the MDL model definition, a meta-version of your model could be (assuming yearly time series):

MODEL

BEHAVIORAL> eq1
TSRANGE 2000 1 2020 1
EQ> ...

BEHAVIORAL> eq2
TSRANGE 2004 1 2020 1
EQ> ...

BEHAVIORAL> eq3
TSRANGE etc...
EQ> ...

END

Having a model such as this, you can have different TSRANGE for different equations. Of course, all equation regressors must be computable (no NAs allowed) inside the (equation-related) TSRANGE, e.g., 2000-2020 for equation eq1, 2004-2020 for eq2, etc..

Hope this will help.