[Feature] Allow for an index.sort.missing option for elasticstack_elasticsearch_index (I checked, did not see it)

[shermanericts]

Is your feature request related to a problem? Please describe.
I am looking at the elasticstack_elasticsearch_index -> sort order field, and I did not see an option for index.sort.missing as described here

Describe the resource you would like to have implemented.
In the elasticstack_elasticsearch_index, I would like an option to handle fields that are missing the above sort fields akin to index.sort.missing. If it already exists, if someone can point me to where and how that would be great.

Describe the solution you'd like
An implementation of the above.

Describe alternatives you've considered
I know that we can provide a nested schema for settings, but it appears deprecated as described in the issue and PR here

Additional context
Add any other context or screenshots about the feature request here.

[github-actions]

🚀 Research Factory Issue Intake has started processing this issue

[github-actions]

🚀 Research Factory Issue Intake has started processing this issue

[github-actions]

🚀 Research Factory Issue Intake has started processing this issue

[github-actions]

Implementation research

Run: 2026-05-11T00:00:00Z · CI run

Edits inside this comment are read as input on the next run but are not preserved verbatim. For
durable feedback, post a comment or edit the issue body.

---

Problem framing

The elasticstack_elasticsearch_index resource exposes sort_field and sort_order for Elasticsearch's index.sort.* static settings, but is missing index.sort.missing and index.sort.mode. All four index.sort.* settings are immutable static settings — they can only be set at index creation time and require destroy-and-recreate to change.

@tobio has confirmed the nested sort block (Approach B) as the implementation spine with these constraints:

1. Keep sort_field and sort_order as deprecated (not removed); add a ConflictsWith validator to prevent mixing old and new attributes.
2. Rather than a plain RequiresReplace on the new sort attribute, investigate RequiresReplaceIf or similar to allow migrating from the legacy attributes to the new form without replacing the index when the sort settings are semantically equivalent.

This run investigated the RequiresReplaceIf feasibility in detail.

Full API surface per the Elastic docs:

Setting	Values	Default
index.sort.field	any field with doc_values (boolean, numeric, date, keyword)	—
index.sort.order	asc, desc	asc
index.sort.missing	_last, _first	_last (always, regardless of order)
index.sort.mode	min, max	min when order=asc; max when order=desc

---

Approaches considered

Approach A — Add flat list attributes sort_missing and sort_mode (mirrors existing pattern)

Add two new top-level list attributes following the same shape as sort_order, whose positions correspond positionally to entries in sort_field.

Schema changes:

"sort_missing": schema.ListAttribute{
    ElementType: types.StringType,
    Description: "How to treat documents missing the sort field. Each entry is '_last' or '_first'. Corresponds positionally to sort_field.",
    Optional:    true,
    PlanModifiers: []planmodifier.List{listplanmodifier.RequiresReplace()},
    Validators: []validator.List{
        listvalidator.ValueStringsAre(stringvalidator.OneOf("_last", "_first")),
    },
},
"sort_mode": schema.ListAttribute{
    ElementType: types.StringType,
    Description: "Which value to pick when a sort field has multiple values. Each entry is 'min' or 'max'. Corresponds positionally to sort_field.",
    Optional:    true,
    PlanModifiers: []planmodifier.List{listplanmodifier.RequiresReplace()},
    Validators: []validator.List{
        listvalidator.ValueStringsAre(stringvalidator.OneOf("min", "max")),
    },
},

Model changes: Add "sort.missing" and "sort.mode" to staticSettingsKeys; add SortMissing types.List and SortMode types.List to tfModel. The reflection-based toIndexSettings() already handles types.List of strings. Add two new cases in compareStaticPlanAndES() and configuredDisplayFromPlanValue() in use_existing.go.

Terraform usage:

resource "elasticstack_elasticsearch_index" "example" {
  name         = "events"
  sort_field   = ["date", "username"]
  sort_order   = ["desc", "asc"]
  sort_missing = ["_last", "_first"]
  sort_mode    = ["max", "min"]
}

Pros:

• Zero breaking change — all existing configs continue to work unchanged.
• Minimal scope: ~60 lines across 5 files.
• No new patterns; existing reflection mechanism handles types.List of strings.

Cons:

• Entrenches the existing design debt: four parallel positionally-coupled lists, with no structural guarantee they have the same length.
• Does not fix the existing sort_field SetAttribute bug (order matters in ES, but a Set does not preserve order).
• Users still cannot see all sort settings for one field in one place.
• No path to addressing the SetAttribute ordering issue.

---

Approach B — Nested sort block with backward-compatible deprecation and RequiresReplaceIf migration support (recommended)

Introduce a new sort attribute as a ListNestedAttribute, where each element groups all per-field sort options together. Keep sort_field and sort_order as deprecated, with ConflictsWith validators. Use a custom plan modifier that suppresses replace when migrating from the legacy attributes to the new form with semantically identical settings.

Schema changes:

// New nested sort attribute
"sort": schema.ListNestedAttribute{
    Description: "Sort configuration for documents within each shard segment. Replaces the deprecated sort_field and sort_order attributes.",
    Optional:    true,
    PlanModifiers: []planmodifier.List{sortMigrationPlanModifier{}},
    Validators: []validator.List{
        listvalidator.ConflictsWith(path.MatchRoot("sort_field"), path.MatchRoot("sort_order")),
    },
    NestedObject: schema.NestedAttributeObject{
        Attributes: map[string]schema.Attribute{
            "field": schema.StringAttribute{Required: true},
            "order": schema.StringAttribute{Optional: true,
                Validators: []validator.String{stringvalidator.OneOf("asc", "desc")}},
            "missing": schema.StringAttribute{Optional: true,
                Validators: []validator.String{stringvalidator.OneOf("_last", "_first")}},
            "mode": schema.StringAttribute{Optional: true,
                Validators: []validator.String{stringvalidator.OneOf("min", "max")}},
        },
    },
},
// Existing attributes — add DeprecationMessage + ConflictsWith
"sort_field": schema.SetAttribute{
    // ... existing ...
    DeprecationMessage: "Use the 'sort' attribute instead. 'sort_field' will be removed in a future release.",
    Validators: []validator.Set{setvalidator.ConflictsWith(path.MatchRoot("sort"))},
    PlanModifiers: []planmodifier.Set{legacySortFieldPlanModifier{}},
},
"sort_order": schema.ListAttribute{
    // ... existing ...
    DeprecationMessage: "Use the 'sort' attribute instead. 'sort_order' will be removed in a future release.",
    Validators: []validator.List{listvalidator.ConflictsWith(path.MatchRoot("sort"))},
    PlanModifiers: []planmodifier.List{legacySortOrderPlanModifier{}},
},

RequiresReplaceIf strategy — private state approach:

The core challenge: sort_field is a SetAttribute, which does not preserve insertion order. When Terraform reads sort_field = ["date", "username"] from state, the Set's internal order is non-deterministic with respect to the original field-to-order correspondence. For multi-field sorts with differing orders, we cannot reconstruct the correct field→order mapping from Terraform state alone.

Solution: Store the ordered sort configuration from Elasticsearch as private state during Read. The ES GET /{index}/_settings API returns sort.field in the originally-configured order. Storing this in private state sidesteps the SetAttribute ordering ambiguity entirely.

The codebase already uses this pattern: api_key stores cluster version in private state during Read and reads it back in a RequiresReplaceIf modifier (internal/elasticsearch/security/api_key/resource.go:106).

Private state stored in Read:

type sortPrivateState struct {
    Fields  []string `json:"fields"`
    Orders  []string `json:"orders"`
}
// stored as JSON under key "sort_config" via resp.Private.SetKey(ctx, "sort_config", data)

Custom sortMigrationPlanModifier (implements planmodifier.List):

The modifier fires on the new sort attribute. It suppresses replace when:

1. sort was null in state (migrating from old representation, not changing the new one)
2. Private state has the ordered sort config from ES
3. Plan's sort[*].field matches private state's ordered fields exactly (same count, same order)
4. Plan's sort[*].order matches private state's orders (accounting for the asc default when not specified in plan)
5. Plan's sort[*].missing and sort[*].mode are all null (new capabilities — any non-null value here means settings are actually changing, so replace required)

If private state is absent (first terraform apply after provider upgrade), the modifier defaults to RequiresReplace = true, protecting correctness at the cost of one forced replace.

The legacySortFieldPlanModifier and legacySortOrderPlanModifier on the deprecated attributes use req.Plan.GetAttribute(ctx, path.Root("sort"), &planSort) to read the new sort from the plan. They suppress replace when sort in plan semantically covers the same settings (using the same private state comparison).

Cross-attribute reads are supported in this codebase: set_unknown_if_access_has_changes.go (internal/elasticsearch/security/api_key/) shows the exact pattern using req.State.GetAttribute(ctx, path.Root("access"), &stateAccess).

Model changes (models.go):

• Add a sortEntryModel struct and Sort types.List field to tfModel.
• toIndexSettings() cannot handle ListNestedAttribute via reflection. Add a pre-processing step: if Sort is non-null, expand to sort.field, sort.order, sort.missing, sort.mode parallel-array entries in the settings map, then skip those keys in the reflection loop.
• Keep "sort.field" and "sort.order" in staticSettingsKeys for the deprecated path.
• Add "sort.missing" and "sort.mode" to staticSettingsKeys.

Read/flatten path (resource.go):

• When reading from ES, check which representation is in state and populate accordingly.
• Also store the ordered sort config from ES in resp.Private.

use_existing.go:

• Add case "sort.missing" and case "sort.mode" branches using stringSliceOrderedFromAny.
• The case "sort" (nested block) is handled at the plan modifier level, not here.

Terraform usage (new):

resource "elasticstack_elasticsearch_index" "example" {
  name = "events"
  sort = [
    { field = "date",     order = "desc" },
    { field = "username", order = "asc"  },
  ]
}

Migration from old to new (no destroy+recreate when semantically equivalent):

# Before (deprecated, still works)
sort_field = ["date", "username"]
sort_order = ["desc", "asc"]

# After (same ES settings, no replace triggered)
sort = [
  { field = "date",     order = "desc" },
  { field = "username", order = "asc"  },
]

If the user adds missing or mode values in the new sort block, replace is required (because those settings weren't configured on the existing index).

Pros:

• Self-documenting: per-field options co-located; length mismatches structurally impossible.
• Covers all four sort settings in one attribute.
• No breaking change for existing users — deprecated attributes remain functional.
• ListNestedAttribute correctly models the ordered nature (unlike the existing SetAttribute for sort_field).
• Plan-time error if user mixes old and new attributes (ConflictsWith).
• RequiresReplaceIf with private state enables seamless migration for the common case (field+order only, no missing/mode).
• Private state pattern is already established in this codebase.

Cons:

• Custom expand/flatten logic required — cannot reuse the reflection mechanism in toIndexSettings().
• More files touched: schema, model, resource read, use_existing, plan modifier (7–8 files, ~200–250 lines).
• If private state is absent on first apply after upgrade, replace is forced once (safe, not catastrophic, but should be documented).
• Adding missing or mode to the new sort block always requires destroy+recreate (unavoidable — those settings couldn't exist on the old index).
• Three plan modifiers (sortMigrationPlanModifier, legacySortFieldPlanModifier, legacySortOrderPlanModifier) with cross-attribute reads add modest complexity.

---

Recommendation

Approach B with the private-state RequiresReplaceIf strategy is the recommended spine.

The nested sort block fixes the design debt (positional coupling, unordered SetAttribute), delivers all four sort settings, and the backward-compatible deprecation avoids forcing migration. The ConflictsWith validator provides a clear plan-time error for mixed usage.

The private state approach for RequiresReplaceIf is the correct mechanism for migration without replace: it stores the ordered sort config from ES during Read, sidesteps the sort_field Set ordering ambiguity, and follows an established codebase pattern (api_key cluster version). The fallback to RequiresReplace = true when private state is absent (first apply after upgrade) is safe and acceptable.

Recommended delivery as a single change:

1. Introduce sort as a ListNestedAttribute covering field, order, missing, and mode.
2. Store ordered sort config in private state during Read.
3. Implement sortMigrationPlanModifier (on sort), legacySortFieldPlanModifier (on sort_field), and legacySortOrderPlanModifier (on sort_order) with cross-attribute reads to suppress replace when migrating with identical settings.
4. Add DeprecationMessage and ConflictsWith to sort_field and sort_order.
5. Add "sort.missing" and "sort.mode" to staticSettingsKeys and expand cases in use_existing.go.
6. Document: adding missing/mode to a migrated sort block requires destroy+recreate; first apply after provider upgrade may force replace once (until private state is populated).

---

Open questions

• Post-upgrade forced replace on first apply: Is the single forced replace (when private state is absent on first apply after provider upgrade) acceptable? Or should we implement a state upgrade path that pre-populates private state from the existing sort_field/sort_order Terraform state values?
• missing/mode default equivalence: Should the modifier treat missing = "_last" (the ES default) as equivalent to null when comparing plan against private state? This would allow users to explicitly write the default and still avoid replace. Likely yes — should be treated as no-op.
• Deprecation timeline: How long before sort_field and sort_order are removed? Major version bump required?
• Cross-list length validator for deprecated path: Should a cross-attribute length validator be added to enforce len(sort_order) == len(sort_field) to prevent silent ES 400s?

---

Out of scope

• Full removal of sort_field and sort_order deprecated attributes — separate change after deprecation period.
• Retroactively fixing the sort_field SetAttribute ordering bug outside of this change.
• Any changes to dynamic settings or non-sort index settings.
• State upgrade functions for migrating private state across provider versions.

---

References

• Elastic Docs: Index sorting settings
• internal/elasticsearch/index/index/schema.go — sort_field (line 240), sort_order (line 249)
• internal/elasticsearch/index/index/models.go — staticSettingsKeys (lines 43–53), tfModel sort fields (lines 119–120), toIndexSettings() (line 353)
• internal/elasticsearch/index/index/use_existing.go — compareStaticPlanAndES sort.field/sort.order cases (lines 179–193)
• internal/elasticsearch/index/index/mapping_modifier.go — custom PlanModifyString pattern (line 31)
• internal/elasticsearch/security/api_key/resource.go — private state pattern: SetKey/GetKey (lines 81–148), clusterVersionPrivateDataKey (line 106)
• internal/elasticsearch/security/api_key/schema.go — RequiresReplaceIf with private state (lines 203–213)
• internal/elasticsearch/security/api_key/set_unknown_if_access_has_changes.go — cross-attribute req.State.GetAttribute pattern (lines 44–74)
• internal/elasticsearch/index/index/acc_test.go — existing sort_field/sort_order assertions (lines 165–166)

🤖 Pipeline metadata

{
  "schema_version": "1.0",
  "recommendation": {
    "spine": "nested-sort-block-with-private-state-requires-replace-if",
    "confidence": "high",
    "approach_index": 1
  },
  "open_questions": [
    {
      "id": "oq-1",
      "text": "Is the single forced replace on first apply after provider upgrade (before private state is populated) acceptable, or should a state upgrade pre-populate private state from existing sort_field/sort_order?",
      "blocking": false
    },
    {
      "id": "oq-2",
      "text": "Should the RequiresReplaceIf modifier treat missing='_last' (ES default) as equivalent to null in the plan, to allow writing explicit defaults without triggering replace?",
      "blocking": false
    },
    {
      "id": "oq-3",
      "text": "What is the deprecation timeline for sort_field and sort_order before removal? Major version bump required?",
      "blocking": false
    },
    {
      "id": "oq-4",
      "text": "Should a cross-attribute length validator be added to the deprecated path to enforce len(sort_order) == len(sort_field)?",
      "blocking": false
    }
  ],
  "affected_capabilities": [
    "elasticstack_elasticsearch_index"
  ],
  "estimated_scope": "medium",
  "references": [
    { "type": "doc", "url": "https://www.elastic.co/docs/reference/elasticsearch/index-settings/sorting" },
    { "type": "path", "path": "internal/elasticsearch/index/index/schema.go" },
    { "type": "path", "path": "internal/elasticsearch/index/index/models.go" },
    { "type": "path", "path": "internal/elasticsearch/index/index/use_existing.go" },
    { "type": "path", "path": "internal/elasticsearch/index/index/mapping_modifier.go" },
    { "type": "path", "path": "internal/elasticsearch/security/api_key/resource.go" },
    { "type": "path", "path": "internal/elasticsearch/security/api_key/schema.go" },
    { "type": "path", "path": "internal/elasticsearch/security/api_key/set_unknown_if_access_has_changes.go" },
    { "type": "path", "path": "internal/elasticsearch/index/index/acc_test.go" }
  ]
}

[tobio]

Approach B — Refactor to a nested sort list-of-objects block

Continue research with this as the recommended approach, persisting with the broken model (implicit matching sort fields by array index) is wrong.

Breaking change: existing users of sort_field and sort_order must migrate.

Avoid the breaking change. Keep sort_field and sort_order for now, but mark them as deprecated. Add a validator to reject configs that set either sort_field or sort_order alongside the new sort nested attribute.