Make DiskManager max_temp_directory_size dynamically adjustable

[Bukhtawar]

Change max_temp_directory_size from u64 to AtomicU64 so the spill disk limit can be adjusted at runtime without requiring exclusive (&mut) access to the DiskManager.

Before this change, set_max_temp_directory_size required &mut self which was unavailable after the DiskManager was shared via Arc (as it always is in production through RuntimeEnv). The only workaround was set_arc_max_temp_directory_size which required Arc::get_mut — always failing when multiple sessions held references.

After this change, set_max_temp_directory_size takes &self and uses an atomic store. Any thread can adjust the limit, and subsequent spill writes immediately see the new value.

Use cases:

• Adaptive spill limits based on available disk space
• Runtime cluster setting changes without restart
• Graceful degradation under disk pressure

The set_arc_max_temp_directory_size method is deprecated but kept for backward compatibility — it now delegates to the &self method.

Performance: AtomicU64::load(Acquire) adds ~1ns per spill write check. Negligible since spill writes take milliseconds.

Which issue does this PR close?

• Closes #.

Rationale for this change

Ensure can be updated in-place based on available disk or in-place disk size increase

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[bharath-techie]

What happens if queries are holding disk above the new configurable limit ? When we change the configuration from 1024 mb to 512 mb, what's the behavior.

[Bukhtawar]

Added tests, thanks for pointing this out

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-execution v53.1.0 (current)
       Built [  24.100s] (current)
     Parsing datafusion-execution v53.1.0 (current)
      Parsed [   0.021s] (current)
    Building datafusion-execution v53.1.0 (baseline)
       Built [  24.158s] (baseline)
     Parsing datafusion-execution v53.1.0 (baseline)
      Parsed [   0.021s] (baseline)
    Checking datafusion-execution v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.246s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure type_method_marked_deprecated: type method #[deprecated] added ---

Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/type_method_marked_deprecated.ron

Failed in:
  method datafusion_execution::disk_manager::DiskManager::set_arc_max_temp_directory_size in /home/runner/work/datafusion/datafusion/datafusion/execution/src/disk_manager.rs:249
  method datafusion_execution::DiskManager::set_arc_max_temp_directory_size in /home/runner/work/datafusion/datafusion/datafusion/execution/src/disk_manager.rs:249

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  51.553s] datafusion-execution