[python-package] [R-package] every test catching an error should check for a specific error message

[jameslamb]

Description

Unit tests that assert that an error occurs should check that a specific error is raised.

Like this in Python:

https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_engine.py#L782-L783

And this in R:

https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/R-package/tests/testthat/test_basic.R#L209-L215

Benefits of this work

• prevents tests from accidentally passing in the presence of an unrelated and unexpected error

Acceptance criteria

• R: every use of expect_error() and expect_warning() in tests should pass argument regexp to catch a specific error/warning
• Python: every use of pytest.raises() and pytest.warns() in tests should pass argument match to catch a specific error/warning, and should catch a more specific exception type than "Exception"

Approach

Do not try to fix all of these at once. Pull requests only changing 1 or a small number at a time are preferred.

Python:

• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_basic.py#L274 ([python-package] Add specific error messages to some test_basic tests #6887)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_basic.py#L281 ([python-package] Add specific error messages to some test_basic tests #6887)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_basic.py#L285 ([python-package] Add specific error messages to some test_basic tests #6887)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_basic.py#L289 ([python-package] Add specific error messages to some test_basic tests #6887)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_basic.py#L685 ([python-package] Add specific error messages to test_basic and test_dask tests #6893)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_basic.py#L689 ([python-package] Add specific error messages to test_basic and test_dask tests #6893)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_dask.py#L1023 ([python-package] Add specific error messages to test_basic and test_dask tests #6893)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_dask.py#L1047 ([python-package] Add specific error messages to test_basic and test_dask tests #6893)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_dask.py#L1133 ( [python-package] Add specific error messages to test_dask tests #6955)
• https://github.com/microsoft/LightGBM/blob/e7c6c4371b5d725902a09a80b4d6c36e432a4381/tests/python_package_test/test_dask.py#L1233
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_engine.py#L2827 ([python-package] Add specific error messages to test_engine #6925)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_engine.py#L2830 ([python-package] Add specific error messages to test_engine #6925)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_engine.py#L2842 ([python-package] Add specific error messages to test_engine #6925)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_engine.py#L2846 ([python-package] Add specific error messages to test_engine #6925)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_engine.py#L2861 ([python-package] Add specific error messages to test_engine #6925)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_engine.py#L3117 ([python-package] Add more specific error messages to test_engine #6926)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_engine.py#L3119 ([python-package] Add more specific error messages to test_engine #6926)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_engine.py#L3153 ([python-package] Add more specific error messages to test_engine #6926)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_plotting.py#L157 ([python-package] Add specific error messages to test_plotting #6927)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_plotting.py#L169 ([python-package] Add specific error messages to test_plotting #6927)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_plotting.py#L187 ([python-package] Add specific error messages to test_plotting #6927)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_plotting.py#L232 ([python-package] Add specific error messages to test_plotting #6927)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_plotting.py#L277 ([python-package] Add specific error messages to test_plotting #6927)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_sklearn.py#L722 ([python-package] Add specific error messages to test_plotting #6927)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_sklearn.py#L1529 ([python-package] Add specific error messages to test_sklearn #6929)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_sklearn.py#L1628 ([python-package] Add specific error messages to test_sklearn #6929)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/tests/python_package_test/test_sklearn.py#L1649 ([python-package] Add specific error messages to test_sklearn #6929)

R

• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/R-package/tests/testthat/test_Predictor.R#L190-L191 ([R-package] Add specific error messages to test_Predictor and test_dataset  #6931)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/R-package/tests/testthat/test_dataset.R#L49 ([R-package] Add specific error messages to test_Predictor and test_dataset  #6931)
• https://github.com/microsoft/LightGBM/blob/6437645c4a0c17046be59e4f57d09952e2e0185f/R-package/tests/testthat/test_dataset.R#L182-L184 ([R-package] Add specific error messages to test_Predictor and test_dataset  #6931)

Notes

How to run the Python tests:

# build lib_lightgbm (just do this once)
cmake -B build -S .
cmake --build build --target _lightgbm -j4

# install Python package (do this every time you change Python code in the library)
sh build-python.sh install --precompile

# run tests
python tests/test_python_package

How to run the R tests:

sh build-cran-package.sh --no-build-vignettes
R CMD INSTALL ./lightgbm*.tar.gz
cd ./R-package/tests
Rscript testthat.R

For other questions on how to set up your development environment, see #6350 (and post questions there about the development process).

[jameslamb]

@nicklamiller thanks so much for your help so far! There are just a few more of these left, could you help us finish this?

[nicklamiller]

@jameslamb sure thing, all of these appear to be fixed except for the final raises in test_dask.py, which I've just opened a PR for. To help add the corr. PR in the issue OP to the remaining unchecked raises by file:

• test_basic.py
  • Remaining raises: [python-package] Add specific error messages to test_basic and test_dask tests #6893
• test_dask.py
  • First two raises: [python-package] Add specific error messages to test_basic and test_dask tests #6893
  • Last raises: [python-package] Add specific error messages to test_dask tests #6955
• test_sklearn.py
  • All raises: [python-package] Add specific error messages to test_sklearn #6929

Also to enforce error messages when testing for errors in the future I believe we can:

• Python package: Add ruff's PT011 to the pyproject.toml file
  • Alternatively, we could enforce the entire flake8-pytest-style (PT) collection of linters, though there are currently 27 related errors so if we want to enforce this we'd have to do a little work before updating the linting config for the sake of CI checks (please see "All PT Errors" below). Plus, some of these are naturally subjective styling so not sure if we necessarily want to enforce them.
• R package: Keeping track of adding this linter to lintr in New linter to enforce specific regexp error message in expect_error r-lib/lintr#2870
  • Doesn't currently exist for testthat linters
  • If maintainers deem useful, I can add that in to lintr and afterwards update in this project and add that to the list of R linters we use

So please let me know if you'd like me to make the immediate proposed change to the python linting config (adding PT011) and if so, if you'd like me to do it as part of this issue or open up a separate issue.

All PT Errors

ruff.....................................................................Failed
- hook id: ruff
- exit code: 1

tests/python_package_test/conftest.py:7:17: PT003 `scope='function'` is implied in `@pytest.fixture()`
  |
7 | @pytest.fixture(scope="function")
  |                 ^^^^^^^^^^^^^^^^ PT003
8 | def missing_module_cffi(monkeypatch):
9 |     """Mock 'cffi' not being importable"""
  |
  = help: Remove implied `scope` argument

tests/python_package_test/conftest.py:14:17: PT003 `scope='function'` is implied in `@pytest.fixture()`
   |
14 | @pytest.fixture(scope="function")
   |                 ^^^^^^^^^^^^^^^^ PT003
15 | def rng():
16 |     return np.random.default_rng()
   |
   = help: Remove implied `scope` argument

tests/python_package_test/conftest.py:19:17: PT003 `scope='function'` is implied in `@pytest.fixture()`
   |
19 | @pytest.fixture(scope="function")
   |                 ^^^^^^^^^^^^^^^^ PT003
20 | def rng_fixed_seed():
21 |     return np.random.default_rng(seed=42)
   |
   = help: Remove implied `scope` argument

tests/python_package_test/test_arrow.py:214:5: PT006 Wrong type passed to first argument of `pytest.mark.parametrize`; expected `tuple`
    |
213 | @pytest.mark.parametrize(
214 |     ["array_type", "label_data"],
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PT006
215 |     [
216 |         (pa.array, [0, 1, 0, 0, 1]),
    |
    = help: Use a `tuple` for the first argument

tests/python_package_test/test_arrow.py:234:5: PT006 Wrong type passed to first argument of `pytest.mark.parametrize`; expected `tuple`
    |
233 | @pytest.mark.parametrize(
234 |     ["array_type", "label_data"],
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PT006
235 |     [
236 |         (pa.array, [False, True, False, False, True]),
    |
    = help: Use a `tuple` for the first argument

tests/python_package_test/test_arrow.py:265:5: PT006 Wrong type passed to first argument of `pytest.mark.parametrize`; expected `tuple`
    |
264 | @pytest.mark.parametrize(
265 |     ["array_type", "weight_data"],
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PT006
266 |     [
267 |         (pa.array, [3, 0.7, 1.5, 0.5, 0.1]),
    |
    = help: Use a `tuple` for the first argument

tests/python_package_test/test_arrow.py:288:5: PT006 Wrong type passed to first argument of `pytest.mark.parametrize`; expected `tuple`
    |
287 | @pytest.mark.parametrize(
288 |     ["array_type", "group_data"],
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PT006
289 |     [
290 |         (pa.array, [2, 3]),
    |
    = help: Use a `tuple` for the first argument

tests/python_package_test/test_arrow.py:311:5: PT006 Wrong type passed to first argument of `pytest.mark.parametrize`; expected `tuple`
    |
310 | @pytest.mark.parametrize(
311 |     ["array_type", "init_score_data"],
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PT006
312 |     [
313 |         (pa.array, [0, 1, 2, 3, 3]),
    |
    = help: Use a `tuple` for the first argument

tests/python_package_test/test_basic.py:284:5: PT012 `pytest.raises()` block should contain a single simple statement
    |
282 |       X2 = rng.uniform(size=(100, 1))
283 |       err_msg = "Both source and target Datasets must be constructed before adding features"
284 | /     with pytest.raises(ValueError, match=err_msg):
285 | |         d1 = lgb.Dataset(X1)
286 | |         d2 = lgb.Dataset(X2)
287 | |         d1.add_features_from(d2)
    | |________________________________^ PT012
288 |       with pytest.raises(ValueError, match=err_msg):
289 |           d1 = lgb.Dataset(X1).construct()
    |

tests/python_package_test/test_basic.py:288:5: PT012 `pytest.raises()` block should contain a single simple statement
    |
286 |           d2 = lgb.Dataset(X2)
287 |           d1.add_features_from(d2)
288 | /     with pytest.raises(ValueError, match=err_msg):
289 | |         d1 = lgb.Dataset(X1).construct()
290 | |         d2 = lgb.Dataset(X2)
291 | |         d1.add_features_from(d2)
    | |________________________________^ PT012
292 |       with pytest.raises(ValueError, match=err_msg):
293 |           d1 = lgb.Dataset(X1)
    |

tests/python_package_test/test_basic.py:292:5: PT012 `pytest.raises()` block should contain a single simple statement
    |
290 |           d2 = lgb.Dataset(X2)
291 |           d1.add_features_from(d2)
292 | /     with pytest.raises(ValueError, match=err_msg):
293 | |         d1 = lgb.Dataset(X1)
294 | |         d2 = lgb.Dataset(X2).construct()
295 | |         d1.add_features_from(d2)
    | |________________________________^ PT012
    |

tests/python_package_test/test_basic.py:983:5: PT018 Assertion should be broken down into multiple parts
    |
981 |     # row-major dataset
982 |     X_row, y = make_blobs(n_samples=1_000, n_features=3, centers=2)
983 |     assert X_row.flags["C_CONTIGUOUS"] and not X_row.flags["F_CONTIGUOUS"]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PT018
984 |     ds_row = lgb.Dataset(X_row, y)
985 |     ds_row_path = tmp_path / "ds_row.txt"
    |
    = help: Break down assertion into multiple parts

tests/python_package_test/test_basic.py:990:5: PT018 Assertion should be broken down into multiple parts
    |
988 |     # col-major dataset
989 |     X_col = np.asfortranarray(X_row)
990 |     assert X_col.flags["F_CONTIGUOUS"] and not X_col.flags["C_CONTIGUOUS"]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PT018
991 |     ds_col = lgb.Dataset(X_col, y)
992 |     ds_col_path = tmp_path / "ds_col.txt"
    |
    = help: Break down assertion into multiple parts

tests/python_package_test/test_dask.py:83:1: PT001 [*] Use `@pytest.fixture` over `@pytest.fixture()`
   |
83 | @pytest.fixture()
   | ^^^^^^^^^^^^^^^^^ PT001
84 | def listen_port():
85 |     listen_port.port += 10
   |
   = help: Remove parentheses

tests/python_package_test/test_dask.py:1024:9: PT012 `pytest.raises()` block should contain a single simple statement
     |
1022 |           )
1023 |
1024 | /         with pytest.raises(AttributeError, match=no_client_attr_msg):
1025 | |             local_model.client
1026 | |             local_model.client_
     | |_______________________________^ PT012
1027 |
1028 |           # should be able to set client after construction
     |

tests/python_package_test/test_dask.py:1048:9: PT012 `pytest.raises()` block should contain a single simple statement
     |
1047 |           local_model = dask_model.to_local()
1048 | /         with pytest.raises(AttributeError, match=no_client_attr_msg):
1049 | |             local_model.client
1050 | |             local_model.client_
     | |_______________________________^ PT012
     |

tests/python_package_test/test_dask.py:1137:13: PT012 `pytest.raises()` block should contain a single simple statement
     |
1135 |                   f"{repr(type(local_model).__name__)} object has no attribute '(client|client_)'"
1136 |               )
1137 | /             with pytest.raises(AttributeError, match=no_client_attr_msg):
1138 | |                 local_model.client
1139 | |                 local_model.client_
     | |___________________________________^ PT012
1140 |
1141 |               tmp_file2 = tmp_path / "model-2.pkl"
     |

tests/python_package_test/test_dask.py:1236:9: PT012 `pytest.raises()` block should contain a single simple statement
     |
1234 |           df = dd.demo.make_timeseries()
1235 |           df = df.map_partitions(f, meta=df._meta)
1236 | /         with pytest.raises(Exception) as info:
1237 | |             lgb.dask._train(client=client, data=df, label=df.x, params={}, model_factory=lgb.LGBMClassifier)
1238 | |             assert "foo" in str(info.value)
     | |___________________________________________^ PT012
     |

tests/python_package_test/test_dask.py:1236:28: PT011 `pytest.raises(Exception)` is too broad, set the `match` parameter or use a more specific exception
     |
1234 |         df = dd.demo.make_timeseries()
1235 |         df = df.map_partitions(f, meta=df._meta)
1236 |         with pytest.raises(Exception) as info:
     |                            ^^^^^^^^^ PT011
1237 |             lgb.dask._train(client=client, data=df, label=df.x, params={}, model_factory=lgb.LGBMClassifier)
1238 |             assert "foo" in str(info.value)
     |

tests/python_package_test/test_dask.py:1365:9: PT012 `pytest.raises()` block should contain a single simple statement
     |
1363 |           # one of the ports mentioned in "machines"
1364 |           error_msg = f"Binding port {open_ports[0]} failed"
1365 | /         with pytest.raises(lgb.basic.LightGBMError, match=error_msg):
1366 | |             with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
1367 | |                 s.bind((workers_hostname, open_ports[0]))
1368 | |                 dask_model.fit(dX, dy, group=dg)
     | |________________________________________________^ PT012
1369 |
1370 |           # The above error leaves a worker waiting
     |

tests/python_package_test/test_dask.py:1381:5: PT006 Wrong type passed to first argument of `pytest.mark.parametrize`; expected `tuple`
     |
1380 | @pytest.mark.parametrize(
1381 |     "dask_est,sklearn_est",
     |     ^^^^^^^^^^^^^^^^^^^^^^ PT006
1382 |     [
1383 |         (lgb.DaskLGBMClassifier, lgb.LGBMClassifier),
     |
     = help: Use a `tuple` for the first argument

tests/python_package_test/test_dask.py:1456:9: PT018 Assertion should be broken down into multiple parts
     |
1454 |         dy = dy.to_dask_array(lengths=True)
1455 |         dy_col_array = dy.reshape(-1, 1)
1456 |         assert len(dy_col_array.shape) == 2 and dy_col_array.shape[1] == 1
     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PT018
1457 |
1458 |         params = {"n_estimators": 1, "num_leaves": 3, "random_state": 0, "time_out": 5}
     |
     = help: Break down assertion into multiple parts

tests/python_package_test/test_engine.py:4797:5: PT018 Assertion should be broken down into multiple parts
     |
4795 | def test_equal_predict_from_row_major_and_col_major_data():
4796 |     X_row, y = make_synthetic_regression()
4797 |     assert X_row.flags["C_CONTIGUOUS"] and not X_row.flags["F_CONTIGUOUS"]
     |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PT018
4798 |     ds = lgb.Dataset(X_row, y)
4799 |     params = {"num_leaves": 8, "verbose": -1}
     |
     = help: Break down assertion into multiple parts

tests/python_package_test/test_engine.py:4804:5: PT018 Assertion should be broken down into multiple parts
     |
4803 |     X_col = np.asfortranarray(X_row)
4804 |     assert X_col.flags["F_CONTIGUOUS"] and not X_col.flags["C_CONTIGUOUS"]
     |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PT018
4805 |     preds_col = bst.predict(X_col)
     |
     = help: Break down assertion into multiple parts

tests/python_package_test/test_sklearn.py:509:39: PT007 Wrong values type in `pytest.mark.parametrize` expected `list` of `tuple`
    |
509 | @pytest.mark.parametrize("estimator", (lgb.LGBMClassifier, lgb.LGBMRegressor, lgb.LGBMRanker))
    |                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PT007
510 | def test_estimators_all_have_the_same_kwargs_and_defaults(estimator):
511 |     base_spec = inspect.getfullargspec(lgb.LGBMModel)
    |
    = help: Use `list` of `tuple` for parameter values

tests/python_package_test/test_sklearn.py:1648:5: PT018 Assertion should be broken down into multiple parts
     |
1646 |       X, y = load_digits(n_class=2, return_X_y=True, as_frame=True)
1647 |       col_names = X.columns.to_list()
1648 | /     assert isinstance(col_names, list) and all(isinstance(c, str) for c in col_names), (
1649 | |         "input data must have feature names for this test to cover the expected functionality"
1650 | |     )
     | |_____^ PT018
1651 |       params = {"n_estimators": 2, "num_leaves": 7}
1652 |       if estimator_class is lgb.LGBMModel:
     |
     = help: Break down assertion into multiple parts

tests/python_package_test/test_sklearn.py:1708:9: PT017 Found assertion on exception `err` in `except` block, use `pytest.raises()` instead
     |
1706 |     except AttributeError as err:
1707 |         # only the exact error we expected to be raised should be raised
1708 |         assert bool(re.search(r"__sklearn_tags__.* should not be called", str(err)))
     |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PT017
1709 |     else:
1710 |         # if no AttributeError was thrown, we must be using scikit-learn>=1.6,
     |

Found 27 errors.
[*] 1 fixable with the `--fix` option (15 hidden fixes can be enabled with the `--unsafe-fixes` option).

[jameslamb]

Oh great, thank you!!! I apologize, looks like I was not doing as good a job of keeping the list in the description up to date as I thought. I just updated all of those and approved + merged #6955.

we could enforce the entire flake8-pytest-style (PT) collection of linters
...
_ let me know if you'd like me to make the immediate proposed change to the python linting config_

Thanks so much for the thorough explanation. I love the idea of introducing ruff's PT011 to help enforce this stuff, and I'd support just immediately doing that (no need to create a new issue).

Looks like it even caught one overly-broad exception that I missed in this issue:

https://github.com/microsoft/LightGBM/blob/e7c6c4371b5d725902a09a80b4d6c36e432a4381/tests/python_package_test/test_dask.py#L1233

I'd support PRs (or even a single PR, if you'd prefer) that introduced PT in the list here:

https://github.com/microsoft/LightGBM/blob/e7c6c4371b5d725902a09a80b4d6c36e432a4381/python-package/pyproject.toml#L141

These look like some great suggestions... for example, only putting with pytest.raises() around the 1 statement that raises the exception is really helpful, and something @StrikerRUS has advised me on before. Would be great to enforce that automatically!

But please add the following to the ignore list

• "PT007 Wrong values type in pytest.mark.parametrize expected list of tuple"
  • I do not really understand this one... a tuple is a little safer because it's immutable
• "PT003 scope='function' is implied in @pytest.fixture()"
  • I like being able to read the code and knowing it's a function-scoped fixture, not needing to remember what the default is

I can add that in to lintr and afterwards update in this project and add that to the list of R linters we use

Thanks for opening that {lintr} issue.

If that idea is accepted and implemented there, I'd support using it here in LightGBM. I understand and agree with some of the concerns @MichaelChirico raised there, but for {lightgbm} (where our expect_error() tests are mostly catching our own first-party error messages), I'd like this linter's help and I'd be willing to to pay the slight maintenance cost of # nolint comments on any expect_error() calls catching errors raised by other libraries.

Totally up to you if working on that in {lintr} interests you!