Skip to content

feat: apply-mask (field) filter allows using another field as the mask#277

Merged
aaron-hopkinson merged 8 commits into
mainfrom
feat/apply-mask-from-field
Apr 20, 2026
Merged

feat: apply-mask (field) filter allows using another field as the mask#277
aaron-hopkinson merged 8 commits into
mainfrom
feat/apply-mask-from-field

Conversation

@aaron-hopkinson
Copy link
Copy Markdown
Contributor

@aaron-hopkinson aaron-hopkinson commented Apr 20, 2026

Description

Alternative implementation of #276, builds on #274 but converts the filter back to a Filter rather than SingleFieldFilter (but keeps alignment to minimise the diff)

Allow masking from a field in the pipeline

The apply_mask filter currently requires an external file (path) to provide the mask. This makes it awkward when the mask (e.g. lsm) is already available as a field in the same pipeline — you need to save it to a separate file first, then reference it.

This PR adds a mask_param option as an alternative to path. When mask_param is set, by default, the mask field is consumed (not returned), but it is possible to force it to be returned with (return_mask: true). All existing mask_value/threshold/rename/param options work exactly the same way.

Usage

input:
  pipe:
    - mars:
        param:
        - sd
        - lsm
    - apply_mask:
        mask_param: lsm   # Use lsm for masking
        mask_value: 0      # Set to NaN where lsm == 0 (sea points)

As a contributor to the Anemoi framework, please ensure that your changes include unit tests, updates to any affected dependencies and documentation, and have been tested in a parallel setting (i.e., with multiple GPUs). As a reviewer, you are also responsible for verifying these aspects and requesting changes if they are not adequately addressed. For guidelines about those please refer to https://anemoi.readthedocs.io/en/latest/

By opening this pull request, I affirm that all authors agree to the Contributor License Agreement.

@aaron-hopkinson aaron-hopkinson requested a review from mc4117 April 20, 2026 10:11
@aaron-hopkinson aaron-hopkinson self-assigned this Apr 20, 2026
@github-project-automation github-project-automation Bot moved this to To be triaged in Anemoi-dev Apr 20, 2026
@github-actions github-actions Bot added tests enhancement New feature or request labels Apr 20, 2026
@aaron-hopkinson aaron-hopkinson changed the title feat/apply-mask-from-field feat: Add mask_with_field filter to mask using another field as the mask Apr 20, 2026
@aaron-hopkinson aaron-hopkinson changed the title feat: Add mask_with_field filter to mask using another field as the mask feat: apply-mask (field) filter allows using another field as the mask Apr 20, 2026
@mc4117
Copy link
Copy Markdown
Member

mc4117 commented Apr 20, 2026

I have tested this PR and checked that the mask works as expected

mc4117
mc4117 previously approved these changes Apr 20, 2026
Copy link
Copy Markdown
Member

@mc4117 mc4117 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passes all checks. LGTM!

@github-project-automation github-project-automation Bot moved this from To be triaged to For merging in Anemoi-dev Apr 20, 2026
@aaron-hopkinson aaron-hopkinson marked this pull request as ready for review April 20, 2026 11:59
@mpvginde
Copy link
Copy Markdown
Contributor

Would it be useful to add the option to not remove the field used for masking?
For instance you would want to mask on lsm but also keep it in the final dataset?

@aaron-hopkinson
Copy link
Copy Markdown
Contributor Author

@mpvginde Added. Is that okay? 🙂

Copy link
Copy Markdown
Member

@mc4117 mc4117 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tested and with the new changes, this transform still works for my use case

@aaron-hopkinson aaron-hopkinson merged commit 76074d4 into main Apr 20, 2026
71 checks passed
@aaron-hopkinson aaron-hopkinson deleted the feat/apply-mask-from-field branch April 20, 2026 14:59
@github-project-automation github-project-automation Bot moved this from For merging to Done in Anemoi-dev Apr 20, 2026
aaron-hopkinson pushed a commit that referenced this pull request Apr 22, 2026
🤖 Automated Release PR

This PR was created by `release-please` to prepare the next release.
Once merged:

1. A new version tag will be created
2. A GitHub release will be published
3. The changelog will be updated

Changes to be included in the next release:
---


##
[0.3.0](0.2.0...0.3.0)
(2026-04-22)


### ⚠ BREAKING CHANGES

* remove use of eval in tabular filters
([#278](#278))

### Features

* `apply-mask` (field) filter allows using another field as the mask
([#277](#277))
([76074d4](76074d4))
* Add support for units compatibility
([#258](#258))
([709dc72](709dc72))
* Deduplicate and reuse code from anemoi utils
([#281](#281))
([c97ac48](c97ac48))
* Remove use of eval in tabular filters
([#278](#278))
([b330a6d](b330a6d))

---
> [!IMPORTANT]
> Please do not change the PR title, manifest file, or any other
automatically generated content in this PR unless you understand the
implications. Changes here can break the release process.
> ⚠️ Merging this PR will:
> - Create a new release
> - Trigger deployment pipelines
> - Update package versions

 **Before merging:**
 - Ensure all tests pass
 - Review the changelog carefully
 - Get required approvals

[Release-please
documentation](https://github.com/googleapis/release-please)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants