Skip to content

Feat/extended news#92

Merged
Firstp1ck merged 85 commits intomainfrom
feat/extended-news
Dec 23, 2025
Merged

Feat/extended news#92
Firstp1ck merged 85 commits intomainfrom
feat/extended-news

Conversation

@Firstp1ck
Copy link
Copy Markdown
Owner

@Firstp1ck Firstp1ck commented Dec 7, 2025

Summary

What's New:

  • News Mode: Complete news feed system with Arch Linux news, security advisories, package updates, and AUR comments. Filter, sort, bookmark, and track read/unread status. Optional startup mode via app_start_mode = news.
  • JSON Caching: Cache AUR and official package JSON responses to disk for change detection and offline date fallback
  • Change Detection: Compare cached vs current JSON to detect package changes (version, maintainer, dependencies, etc.) and display in news content
  • Background Retry Queue: Failed package date fetches are queued and retried sequentially with exponential backoff (10s, 20s, 40s), up to 3 attempts per package
  • Background Continuation: After initial limit (50 items), continue fetching all remaining items in background and stream to UI at 1 per second
  • Package Date Fetching: Fetches package update dates from archlinux.org JSON endpoints with fallback to cached data, handles multiple repo/arch combinations, and distinguishes HTTP status codes (404 vs 429/5xx)
  • Date Parsing: Handles RFC3339 format with milliseconds, RSS dates, and normalizes to YYYY-MM-DD for consistent sorting
  • AUR Package Allocation: AUR packages get dedicated allocation (half of limit) to ensure representation alongside official packages
  • Reliability: Rate limiting, circuit breakers, and error recovery prevent IP blocking from archlinux.org (404s don't trigger circuit breaker)
  • Performance: Multi-layer caching (15min memory, 14 days disk) reduces network requests
  • Code Quality: Improved clippy allow comments, reduced function complexity, added CodeQL workflow, improved documentation with rustdoc comments and inline explanations
  • Refactoring: Modularized large source files into organized submodules (sources/feeds, sources/news, events/modals/tests, ui/results/title, app_state, workers)
  • Logging: Promoted important operational messages from DEBUG to INFO level for better visibility
  • i18n: Made config directory alert detection language-agnostic using path patterns instead of hardcoded strings

Bug Fixes (to existing code in main branch):

  • Fixed updates window text alignment when package names wrap
  • Fixed options menu key bindings to match display order
  • Fixed installed_packages.txt export to respect installed_packages_mode setting
  • Fixed alert title showing "Connection issue" instead of "Configuration Directories" for config directory messages after package removal
  • Fixed Shift+Tab keybind to also work in News mode (previously only worked in Package mode)
  • Fixed overflow handling in scroll calculations with proper clamping to prevent incorrect scroll positions

Type of change

  • feat (new feature)
  • fix (bug fix)
  • refactor (no functional change)
  • perf (performance)
  • test (add/update tests)
  • chore (build/infra/CI)
  • style (formatting, code style)
  • ui (visual/interaction changes)

How to test

Basic Tests:

cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings
cargo check
cargo test -- --test-threads=1

News Mode:

  1. Launch Pacsea, switch to News mode (or set app_start_mode = news)
  2. Verify news items load (Arch news, advisories, updates, AUR comments)
  3. Test filters, sorting, read/unread tracking, and bookmarks
  4. Check loading messages appear on first launch
  5. Test Shift+Tab cycles through news sort modes (Date↓, Date↑, Title, Source+Title, Severity+Date, Unread+Date)
  6. Verify background continuation streams additional items after initial 50 (check logs for "continuation worker")
  7. Verify package update dates are correct (not showing today's date when network fails)
  8. Check news content shows JSON change descriptions for AUR and official packages
  9. Verify AUR packages appear even when official packages fill the limit

Reliability:

  • Verify no 429 errors in logs (rate limiting working)
  • Test cached content loads faster on subsequent views
  • Verify circuit breaker activates on failures and recovers

Bug Fixes:

  • See "Bug Fixes (to existing code in main branch)" section above

Checklist

  • Code compiles, formats, and passes clippy
  • All tests pass
  • New functions have rustdoc comments
  • No unwrap() or expect() in non-test code
  • Changes respect --dry-run flag
  • Code degrades gracefully if tools unavailable
  • No breaking changes

Notes for reviewers

Configuration:

  • app_start_mode: "news" to start in News mode (default: "package")
  • news_filter_*: Toggle filters for Arch news, advisories, updates, AUR updates/comments
  • news_max_age_days: Maximum age filter (default: unlimited)

New Files:

  • news_feed.json, news_content_cache.json, news_seen_pkg_updates.json, news_seen_aur_comments.json, news_recent_searches.json, news_bookmarks.json, news_read_urls.json
  • cache/aur_json/ - Cached AUR package JSON responses for change detection
  • cache/official_json/ - Cached official package JSON responses for change detection and date fallback

Technical Highlights:

  • Rate Limiting: Serialized archlinux.org requests (1 at a time) with exponential backoff (2s→4s→8s→16s, max 60s)
  • Circuit Breaker: Per-endpoint failure detection prevents cascading failures (404s don't trigger circuit breaker)
  • Conditional Requests: ETag/Last-Modified headers for efficient updates
  • Timeouts: 15s connect, 30s total for news; 5s for AUR comments; 2s for package dates
  • Fallback: Uses checkupdates when database sync fails (Landlock restrictions)
  • UI: Multi-line keybinds, improved alignment, better menu organization
  • Code Quality: Enhanced clippy comments with line counts, reduced complexity via helper functions and type aliases, CodeQL workflow
  • Refactoring: Split large files (2981-line feeds.rs, 1731-line news.rs, 1689-line tests.rs, 1448-line title.rs) into modular subdirectories; extracted alert message type detection and formatting into helper functions; changed PathBuf parameters to Path for better flexibility; removed unused _url parameter from fetch_package_page_sync
  • Documentation: Added comments explaining intentionally unused parameters; added rustdoc comments to test functions; documented permit dropping consequences in rate_limit_archlinux; documented 500ms debounce delay rationale
  • i18n: Added translation keys for config directory alerts (en-US, de-DE, hu-HU); made detection language-agnostic using path pattern matching

Breaking changes

None. All changes are backward compatible.

- Added support for a news feed mode, allowing users to view and manage news items.
- Introduced new AppMode enum to toggle between package management and news views.
- Enhanced UI components to render news details, bookmarks, and search history.
- Implemented caching for news content and recent searches to improve performance.
- Updated localization files for new UI elements and labels.
- Refactored event handling to accommodate news-specific interactions.
- Added detailed comments and documentation for various structs and functions to improve code clarity and maintainability.
- Introduced type aliases for result handling in multiple modules to standardize error management.
- Organized module imports and improved the structure of UI and application state components for better readability.
- Enhanced logging utilities and preflight modal rendering with additional context and helper functions.
- Implemented a new news search history feature, allowing users to view and manage recent news searches.
- Added localization support for new UI elements related to news history and bookmarks.
- Refactored application state to include new properties for managing news filters and recent searches.
- Updated UI components to render news history and bookmarks, improving user experience in news mode.
- Enhanced event handling to support news-specific interactions and improved performance through caching.
@Firstp1ck Firstp1ck self-assigned this Dec 7, 2025
Firstp1ck and others added 3 commits December 8, 2025 12:26
- feat: include installed updates and aur comments with seen-state dedupe
- feat: persist cached feed and last-seen maps plus filters/loading ui
- feat: render aur comment/news content with link styling and summaries
@Firstp1ck Firstp1ck marked this pull request as ready for review December 8, 2025 22:56
Copilot AI review requested due to automatic review settings December 8, 2025 22:56
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a comprehensive News mode feature that provides an alternative UI experience focused on aggregating Arch Linux news, security advisories, package updates, and AUR comments. The implementation follows existing codebase patterns for state management, persistence, and error handling.

Key changes include:

  • New AppMode enum to toggle between Package and News modes
  • Aggregated news feed combining multiple sources (Arch RSS, security advisories, package updates, AUR comments)
  • Persistent state for bookmarks, read tracking, and search history specific to news mode
  • News-specific UI components with filtering, sorting, and content rendering
  • Background workers for fetching news content and aggregating feed items

Reviewed changes

Copilot reviewed 106 out of 107 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/util/pacman.rs Added documentation for Result type alias
src/util/curl.rs Added comprehensive documentation for Result type alias
src/ui/results/title.rs Added field-level documentation for i18n strings and layout structs
src/ui/results/sort_menu.rs Extended sort menu to support news-specific sorting modes with checkmarks
src/ui/results/mod.rs Added news results rendering with filter chips, source labels, and keyword highlighting
src/ui/results/list.rs Added field documentation for package list status struct
src/ui/results/dropdowns.rs Updated panels/options menus to show news-specific entries in News mode
src/ui/modals/renderer.rs Added field documentation for modal context structs
src/ui/modals/preflight/tabs/summary.rs Added documentation for bit flags and incomplete state tracking
src/ui/modals/preflight/tabs/sandbox.rs Added field documentation for sandbox tab context and state
src/ui/modals/preflight/tabs/mod.rs Added module-level documentation comments
src/ui/modals/preflight/tabs/files.rs Added field documentation for file display and summary structs
src/ui/modals/preflight/tabs/deps.rs Added field documentation for dependency statistics struct
src/ui/modals/preflight/mod.rs Added module-level documentation comments
src/ui/modals/preflight/helpers/widget.rs Added field documentation for paragraph builder
src/ui/modals/preflight/helpers/mod.rs Added module-level documentation comments
src/ui/modals/preflight/helpers/extract.rs Added field documentation for preflight fields struct
src/ui/modals/preflight/header.rs Added field documentation for tab header context
src/ui/modals/news.rs Added documentation for modal sizing constants
src/ui/modals/mod.rs Added module-level documentation comments
src/ui/modals/announcement.rs Improved constant documentation with inline comments
src/ui/middle/search.rs Updated search title to show "News search" in News mode
src/ui/middle/recent.rs Added news search history rendering function
src/ui/middle/mod.rs Added news-specific middle row rendering with bookmarks pane
src/ui/helpers/tests.rs Added test for news history filtering
src/ui/helpers/query.rs Skip recent preview trigger in News mode
src/ui/helpers/mod.rs Added module documentation for logging utilities
src/ui/helpers/logging.rs Added field documentation for ChangeLogger
src/ui/helpers/filter.rs Updated filtering to use news recent values in News mode
src/ui/details/pkgbuild_highlight.rs Added documentation for PKGBUILD cache structures
src/ui/details/mod.rs Added news details rendering with content highlighting and link styling
src/ui/details/footer.rs Added news-specific keybind footer rendering
src/ui.rs Added module documentation and routed to news details rendering
src/theme/types.rs Added news filter settings and start_in_news configuration
src/theme/settings/parse_settings.rs Added parsing for news filter settings
src/theme/settings/mod.rs Added documentation for settings cache structure
src/theme/mod.rs Re-exported news-specific save functions
src/theme/config/skeletons.rs Added news settings to default config template
src/theme/config/settings_save.rs Added save functions for news filter settings
src/theme/config/settings_ensure.rs Added news settings to ensure function
src/theme/config.rs Re-exported news save functions
src/state/types.rs Added comprehensive news-related types (AppMode, NewsFeedItem, etc.)
src/state/modal.rs Added field documentation for modal enums and structs
src/state/app_state/mod.rs Added extensive news state fields and refresh logic
src/state/app_state/defaults.rs Added news feed state initialization with cache loading
src/state/app_state/default_impl.rs Integrated news state into default AppState construction
src/sources/status/mod.rs Added module documentation comments
src/sources/pkgbuild.rs Improved rate limiter documentation
src/sources/news.rs Added comprehensive news content fetching and HTML parsing
src/sources/mod.rs Added advisories module and re-exported news functions
src/sources/feeds.rs New aggregated news feed fetcher with deduplication
src/sources/details.rs Added field documentation for parsed fields struct
src/sources/comments.rs Added stable comment ID extraction
src/sources/advisories.rs New security advisories fetcher
src/main.rs Added documentation for logging utilities
src/logic/summary.rs Added field documentation for post-summary data
src/logic/preflight/mod.rs Improved preflight outcome documentation
src/logic/preflight/metadata.rs Added field documentation for official metadata
src/logic/preflight/command.rs Added field documentation for command error enum
src/logic/files/pkgbuild_fetch.rs Improved rate limiter documentation
src/logic/files/pkgbuild_cache.rs Added comprehensive cache structure documentation
src/logic/deps/reverse.rs Added field documentation for reverse dependency structs
src/install/remove.rs Added field documentation for spawn context
src/install/patterns.rs Added documentation for security section priority enum
src/install/mod.rs Added module-level documentation comments
src/index/mod.rs Added field and module documentation
src/i18n/resolver.rs Added field documentation for locale resolver
src/i18n/loader.rs Added field documentation for locale loader
src/events/utils.rs Added news selection movement and URL update functions
src/events/search/normal_mode.rs Updated navigation to support News mode
src/events/search/mod.rs Added module documentation comments
src/events/search/insert_mode.rs Updated input handling for News mode
src/events/search/helpers.rs Updated pane navigation for News mode
src/events/recent.rs Updated recent list handling for News mode
src/events/preflight/modal.rs Added field documentation for tab parameter structs
src/events/preflight/keys/context.rs Added field documentation for key context structs
src/events/mouse/panes.rs Updated results pane mouse handling for News mode
src/events/mouse/modals/simple.rs Added documentation for news modal mouse handler
src/events/mouse/menus.rs Updated menus for News mode with age toggle
src/events/mouse/filters.rs Added news filter click handling with tests
src/events/mod.rs Made utils module public for news functions
src/app/terminal.rs Added Result type documentation
src/app/runtime/workers/mod.rs Added module documentation for workers

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Firstp1ck and others added 14 commits December 9, 2025 00:21
- Added functionality to clear stale content and reset loading state when loading a news bookmark without cached content.
- Updated event handling to ensure loading flags are managed correctly during content updates.
- Introduced tests to verify behavior for mismatched URLs and stale content scenarios.
- Introduced a new mechanism to persist and manage read news item IDs, allowing users to track which items have been read at a feed level.
- Added key bindings for marking news feed items as read, unread, and toggling their read state.
- Implemented UI updates to reflect read/unread status in the news feed, improving user experience.
- Enhanced application state to include new properties for managing read news IDs and their persistence.
- Updated tests to ensure correct functionality of the new read tracking features and key bindings.
- Added new settings to toggle visibility of AUR updates and comments in the news feed.
- Implemented UI components to manage AUR update filters independently from other news items.
- Updated application state and settings to persist user preferences for news filters.
- Enhanced event handling to support new AUR filter interactions and ensure correct rendering in the UI.
- Introduced settings for configuring the startup news popup, including options for displaying Arch news, advisories, AUR updates, AUR comments, and package updates.
- Implemented a modal for users to set their preferences for the startup news display.
- Enhanced application state to track whether the startup news configuration has been completed and to manage the maximum age of news items.
- Updated localization files to support new UI elements and settings related to the startup news popup.
- Improved event handling to trigger news fetching based on user configuration during application startup.
- Updated the news fetching functions to use shorter timeouts (10s connect, 15s max) to prevent blocking on slow or unreachable servers.
- Refactored the curl calls to include timeout arguments for better performance and reliability in fetching news content.
- Added multiple tests for news fetching, including validation of read/unread states, JSON serialization, and filtering logic for already-read items.
- Implemented tests to ensure correct handling of news items based on user-defined limits and cutoff dates.
- Enhanced error handling and message formatting for news content fetching failures.
- Improved RSS parsing tests to validate behavior with various input scenarios, including missing tags and limit enforcement.
- Introduced new sorting options for news items based on severity and unread status, allowing users to prioritize critical advisories and unread items.
- Implemented a severity ranking system to facilitate sorting, ensuring higher severities are displayed first.
- Updated the application state and UI components to support the new sorting features, including adjustments to the sort menu and rendering logic.
- Added comprehensive tests to validate the new sorting functionalities and ensure correct behavior across various scenarios.
- Added a timeout mechanism for news content fetching to prevent indefinite loading states, with a 6-second limit.
- Introduced detailed logging for news content requests and responses, including success and failure cases.
- Updated application state to track the start time of content loading for better timeout management.
- Enhanced event handling to provide user feedback on loading failures and ongoing requests.
- Implemented a caching mechanism for news article content to improve loading performance and user experience.
- Added functionality to persist and manage the news content cache, ensuring that error messages are filtered out.
- Updated UI components to reflect news loading states and handle interactions for displaying news items.
- Enhanced event handling to support new news button interactions and ensure correct rendering based on application mode.
- Improved localization for news-related UI elements and added tests to validate new caching and UI functionalities.
- Improved news content caching mechanism for better loading performance and persistence.
- Added a loading timeout for news content fetching to prevent indefinite loading states.
- Introduced a startup configuration modal for news display preferences on first launch.
- Enhanced sorting options for news items based on severity and unread status.
- Updated UI components and application state to support new features and improve user experience.
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 134 out of 144 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- docs: document why _pkg parameters are unused in scan command builders
- docs: document why _do_* parameters are unused in spawn_aur_scan_for_with_config
- docs: add inline comments explaining API consistency for unused parameters
- docs: clarify that underscore prefix suppresses Rust/clippy warnings intentionally
@Firstp1ck
Copy link
Copy Markdown
Owner Author

@summoner001 As this is a massive change, I would like to have a human reviewer/tester.
The target for this is that you only test the new "News Management" section opened by the Option Dropdown.

I want your honest opinion:

  • I know there are long waiting times for News loading (>1 min).

  • Otherwise, please test the new process as much as you can. Primary is the User Experience!

  • You do not look at the Code itself, if you are not comfortable.

This would be really appreciated!

If you need to setup development environment, check Dev Setup, Step by step

- refactor: Create FastSourcesConfig struct to reduce fetch_fast_sources arguments from 8 to 1
- refactor: Add PrepareFetchContextReturn type alias to reduce type complexity
- refactor: Extract helper functions in insert_mode.rs to reduce cognitive complexity
- style: Fix documentation backticks for doc_markdown clippy compliance
- refactor: Remove unnecessary clippy allow attributes after fixes
- refactor: add line count information to all clippy::too_many_lines allow comments
- refactor: add explanation comments to allow attributes that were missing them
- refactor: remove unnecessary clippy::too_many_lines allow from functions under 150 line threshold
- fix: Move translation keys from app.modals.loading to app.loading in all locale files
- refactor: Promote important operational messages from DEBUG to INFO level
- refactor: Update update check completion message to INFO
- refactor: Update index save and status messages to INFO
- refactor: Update fallback mechanism messages to INFO
- refactor: split sources/feeds.rs into modular feeds/ directory structure
- refactor: split sources/news.rs into modular news/ directory structure
- refactor: split events/modals/tests.rs into organized tests/ subdirectory
- refactor: split ui/results/title.rs into modular title/ subdirectory
- refactor: extract app_state constants, methods, and tests into separate modules
- refactor: extract news and updates workers from auxiliary.rs into dedicated modules
- refactor: improve code organization and maintainability through modular structure
- fix: change alert title from "Connection issue" to "Configuration Directories" for config directory messages
- refactor: make config directory detection language-agnostic using path patterns instead of hardcoded English strings
- change: add translation keys for config directory alert in en-US, de-DE, and hu-HU locales
- refactor: extract message type detection and formatting into helper functions to reduce complexity
- change: use mauve color (informational) instead of red (error) for config directory alerts
- change: improve message formatting to display config directories line by line for better readability
@summoner001
Copy link
Copy Markdown
Collaborator

But I don't know how to coding/programming :(

@Firstp1ck
Copy link
Copy Markdown
Owner Author

You don't have to look at the code. 😄

You would only have to follow the step-by-step instructions I gave. You don't have to, of course.

This would be the last test, I would create a new release afterward.

@summoner001
Copy link
Copy Markdown
Collaborator

Ok. I'll give it a try on the weekend.
But I might only have time on Sunday. I have to go buy gifts.

- feat: implement JSON caching for AUR and official package data
- feat: add change detection comparing cached vs current JSON responses
- feat: display detected changes in news content for AUR and official packages
- feat: add background retry queue for failed package date fetches
- feat: return news feed immediately, process retries in background task
- feat: background retries update JSON cache for correct dates on next view
- feat: conservative retry: sequential processing with exponential backoff (10s, 20s, 40s)
- feat: failed retries go back to end of queue (up to 3 attempts per package)
- feat: background continuation fetching after initial limit reached
- feat: stream additional news items at 1 per second in background
- feat: no limit on continuation - fetches all available items from all sources
- fix: extract last_update from top-level JSON (not nested in pkg object)
- fix: parse date formats with milliseconds (e.g., 2025-12-15T19:30:14.422Z)
- fix: fall back to cached JSON for dates when network requests fail
- fix: detect actual HTTP status codes in curl_json (404, 429, 5xx etc.)
- fix: try multiple repo/arch combinations for package JSON (core/extra, x86_64/any)
- fix: handle packages in "any" arch correctly (e.g., base, alsa-firmware)
- fix: don't count 404s as rate limiting errors (prevents circuit breaker trips)
- fix: always process AUR packages even when official packages fill the limit
- fix: give AUR packages their own allocation (half of limit) for representation
- fix: normalize Arch news dates to YYYY-MM-DD format for correct sorting
- fix: Shift+Tab now cycles through news sort modes in News management window
- refactor: extract network fetching logic from fetch_news_content into separate function
- change: simplify package update title to just show package name
- change: only show version summary if there's an actual version change
- change: remove updates_versions filtering - now check all installed packages
@Firstp1ck Firstp1ck requested a review from Copilot December 23, 2025 22:15
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 116 out of 197 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

src/events/search/normal_mode.rs:1

  • Potential integer overflow when multiplying retry_after * 1000. If the server returns a very large Retry-After value (e.g., 2^53 seconds), this multiplication could overflow. Consider using checked_mul or capping retry_after before multiplication.
use crossterm::event::{KeyCode, KeyEvent, KeyModifiers};

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/sources/feeds/tests.rs
Comment thread src/sources/feeds/rate_limit.rs
Comment thread src/sources/feeds/helpers.rs Outdated
Comment thread src/events/utils.rs
Comment thread src/events/mouse/modals/simple.rs
Comment thread src/index/mirrors.rs
- docs: add rustdoc comment to sort_news_items_orders_by_date_desc test
- docs: document permit dropping consequences in rate_limit_archlinux
- refactor: change PathBuf parameters to Path for better flexibility
- docs: document 500ms debounce delay rationale
- fix: improve overflow handling in scroll calculations with proper clamping
- refactor: remove unused _url parameter from fetch_package_page_sync
@Firstp1ck Firstp1ck requested a review from Copilot December 23, 2025 22:32
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 117 out of 198 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…locale file

- created command for cursor ide
@Firstp1ck Firstp1ck merged commit fa94dab into main Dec 23, 2025
5 checks passed
@Firstp1ck Firstp1ck deleted the feat/extended-news branch December 23, 2025 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants