Add reconnect logic for handling update failures in DbusService (#250)

* Add reconnect logic for AhoyDTU/OpenDTU after failed update attempts

* Add reconnect logic tests for DbusService to handle update failures

* Refactor DbusService tests to reset meter data and clean up comments

* Implement reconnect logic in DbusService to handle update failures and manage status codes

* Refactor ReconnectLogicTest to improve setup and add comprehensive tests for reconnect behavior and status code handling

* Enhance reconnect logic in DbusService and configuration files to improve error handling and retry mechanisms

* Add configuration options for retry logic in README and config.example

* Refactor status code handling in DbusService to use constants and improve update logic

* Add error handling modes and update logic in DbusService and tests

- Introduced ErrorMode configuration to switch between "retrycount" and "timeout" modes.
- Updated DbusService to handle error states based on the selected mode.
- Enhanced unit tests to cover new timeout behavior and ensure correct status code handling.

* Refactor DbusService update logic to ensure _refresh_data is called in normal operation and add unit tests for successful update scenarios

* Add error handling modes to README and config.example for improved error management

* Enhance error handling configuration in DbusService and tests

- Added error handling mode and related configuration options to config.example.
- Refactored DbusService to initialize and load error handling properties from configuration.
- Updated unit tests to verify correct reading of error handling configuration values.

* Refactor error handling logic in DbusService to fix the retry condition

* Refactor error handling configuration in DbusService to use constants and update several places accordingly to the code review

* minor changes

Author: 0x7878

README.md

@@ -15,6 +15,7 @@
       - [Default options](#default-options)
       - [Inverter options](#inverter-options)
       - [Template options](#template-options)
+      - [Error Handling Modes](#error-handling-modes)
     - [Service names](#service-names)
     - [Videos how to install](#videos-how-to-install)
     - [Use Cases](#use-cases)
@@ -122,12 +123,15 @@ Within the project there is a file `/data/dbus-opendtu/config.ini`. Most importa
 | useYieldDay | send YieldDay instead of YieldTotal. Set this to 1 to prevent VRM from adding the total value to the history on one day. E.g. if you don't start using the inverter at 0. |
 | ESP8266PollingIntervall |  For ESP8266 reduce polling intervall to reduce load, default 10000ms|
 | Logging | Valid options for log level: CRITICAL, ERROR, WARNING, INFO, DEBUG, NOTSET, to keep logfile small use ERROR or CRITICAL |
- MaxAgeTsLastSuccess | Maximum accepted age of ts_last_success in Ahoy status message. If ts_last_success is older than this number of seconds, values are not used.  Set this to < 0 to disable this check.                                    |
+| MaxAgeTsLastSuccess | Maximum accepted age of ts_last_success in Ahoy status message. If ts_last_success is older than this number of seconds, values are not used.  Set this to < 0 to disable this check.                                    |
 | DryRun | Set this to a value different to "0" to prevent values from being sent. Use this for debugging or experiments. |
 | Host | IP or hostname of ahoy or OpenDTU API/web-interface |
 | HTTPTimeout | Timeout when doing the HTTP request to the DTU or template. Default: 2.5 sec |
 | Username | use if authentication required, leave empty if no authentication needed |
 | Password | use if authentication required, leave empty if no authentication needed |
+| MinRetriesUntilFail | Minimum number of consecutive update failures before entering error state (StatusCode=10, zero values). Default is 3. |
+| RetryAfterSeconds | If AhoyDTU/OpenDTU is not reachable, try to reconnect after this many seconds. Default is 120. |
+| ErrorMode | Error handling mode: `retrycount` (default, error after N failures) or `timeout` (error after a time period without success). See section below for details. |
 
 *1: Please assure that the order is correct in the DTU, we can only extract the first one in a row.
 
@@ -182,6 +186,54 @@ This applies to each `TEMPLATE[X]` section. X is the number of Template starting
 
 *4: Path in JSON: use keywords and array index numbers separated by `/`. Example (compare [tasmota_shelly_2pm.json](docs/tasmota_shelly_2pm.json)): `StatusSNS/ENERGY/Current/0` fetches dictionary (map) entry `StatusSNS` containting an entry `ENERGY` containing an entry `Current` containing an array where the first element (index 0) is taken.
 
+---
+
+#### Error Handling Modes
+
+The error handling behavior of dbus-opendtu can be configured using the `ErrorMode` and `ErrorStateAfterSeconds` options in your configuration file. This allows you to choose between two flexible strategies for handling communication errors with your DTU (Data Transfer Unit):
+
+##### 1. `retrycount` Mode (Default)
+- **Behavior:**
+  - The system will attempt to update data from the DTU on every cycle.
+  - If a number of consecutive update attempts fail (as set by `MinRetriesUntilFail`), the system enters an error state:
+    - All DBus values are set to zero.
+    - The DBus `StatusCode` is set to 10 (error).
+  - After waiting for `RetryAfterSeconds`, the system will attempt to reconnect and recover.
+- **Configuration:**
+  - `ErrorMode=retrycount`
+  - `MinRetriesUntilFail=3` (default)
+  - `RetryAfterSeconds=120` (default)
+
+##### 2. `timeout` Mode
+- **Behavior:**
+  - The system always attempts to reconnect and refresh data every `RetryAfterSeconds`.
+  - Zero values and error state are only set if the time since the last successful update exceeds `ErrorStateAfterSeconds`.
+  - This means the system will keep trying to reconnect, but will only show an error after a defined timeout period has passed without success.
+- **Configuration:**
+  - `ErrorMode=timeout`
+  - `ErrorStateAfterSeconds=600` (for example, 10 minutes)
+  - `RetryAfterSeconds=120` (default)
+
+##### Example Configuration
+
+```
+# Error handling mode: "retrycount" (default, as before) or "timeout" (after a time period)
+ErrorMode=timeout
+# For "timeout" mode:
+ErrorStateAfterSeconds=600
+# For both modes:
+RetryAfterSeconds=120
+MinRetriesUntilFail=3
+```
+
+##### Summary Table
+| Mode        | When are zero values set?                | When is reconnect attempted?         |
+|-------------|------------------------------------------|--------------------------------------|
+| retrycount  | After N consecutive failures             | After `RetryAfterSeconds`            |
+| timeout     | After `ErrorStateAfterSeconds` timeout   | Always, every `RetryAfterSeconds`    |
+
+Choose the mode that best fits your reliability and error reporting needs. For most users, the default `retrycount` mode is sufficient. Use `timeout` mode if you want to avoid error states for short outages and only show errors after a longer period without successful updates.
+
 ### Service names
 
 The following servicenames are supported:

config.example

@@ -27,6 +27,19 @@ Logging=ERROR
 # Set this to < 0 to disable this check.
 MaxAgeTsLastSuccess=600
 
+# Error handling mode: "retrycount" (default, as before) or "timeout" (after a time period)
+ErrorMode=retrycount
+
+# If AhoyDTU/OpenDTU is not reachable, try to reconnect after this many seconds Default is 120.
+RetryAfterSeconds=120
+
+# Minimum number of consecutive update failures before entering error state (StatusCode=10, zero values). Default is 3.
+MinRetriesUntilFail=3
+
+# This configuration option is used for the "timeout" mode.
+# The value should be specified in seconds (e.g., 600 seconds for 10 minutes).
+ErrorStateAfterSeconds=600 
+
 # if this is not 0, then no values are actually sent via dbus to vrm/venus.
 DryRun=0
 
@@ -39,6 +52,7 @@ HTTPTimeout=2.5
 Username =
 Password =
 
+
 ### Only needed for OpenDTU and ahoy
 # Phase: Either L1, L2, L3 or 3P for 3 phase HMT series, if unsure use L1 
 # AcPosition 0=AC input 1; 1=AC output; 2=AC input 2

constants.py

@@ -7,6 +7,15 @@
 DTUVARIANT_TEMPLATE = "template"
 PRODUCTNAME = "henne49_dbus-opendtu"
 CONNECTION = "TCP/IP (HTTP)"
+MODE_TIMEOUT = "timeout"
+MODE_RETRYCOUNT = "retrycount"
+
+# Status codes for the DTU
+STATUSCODE_STARTUP = 0
+STATUSCODE_RUNNING = 7
+STATUSCODE_STANDBY = 8
+STATUSCODE_BOOTLOADING = 9
+STATUSCODE_ERROR = 10
 
 
 VICTRON_PATHS = {

dbus_service.py

@@ -79,6 +79,14 @@ def __init__(
         self.meter_data = None
         self.dtuvariant = None
 
+        # Initialize error handling properties
+        self.error_mode = None
+        self.retry_after_seconds = 0
+        self.min_retries_until_fail = 0
+        self.error_state_after_seconds = 0
+        self.failed_update_count = 0
+        self.reset_statuscode_on_next_success = False
+
         if not istemplate:
             self._read_config_dtu(actual_inverter)
             self.numberofinverters = self.get_number_of_inverters()
@@ -118,7 +126,7 @@ def __init__(
         self._dbusservice.add_path("/Serial", self._get_serial(self.pvinverternumber))
         self._dbusservice.add_path("/UpdateIndex", 0)
         # set path StatusCode to 7=Running so VRM detects a working PV-Inverter
-        self._dbusservice.add_path("/StatusCode", 7)
+        self._dbusservice.add_path("/StatusCode", constants.STATUSCODE_RUNNING)
 
         # If the Servicname is an (AC-)Inverter, add the Mode path (to show it as ON)
         # Also, we will set different paths and variables in the _update(self) method.
@@ -143,7 +151,7 @@ def __init__(
                 writeable=True,
                 onchangecallback=self._handlechangedvalue,
             )
-        
+
         self._dbusservice.register()
 
         self.polling_interval = self._get_polling_interval()
@@ -214,6 +222,7 @@ def _read_config_dtu(self, actual_inverter):
         self.pollinginterval = int(get_config_value(config, "ESP8266PollingIntervall", "DEFAULT", "", 10000))
         self.meter_data = 0
         self.httptimeout = get_default_config(config, "HTTPTimeout", 2.5)
+        self._load_error_handling_config(config)
 
     def _read_config_template(self, template_number):
         config = self._get_config()
@@ -267,6 +276,15 @@ def _read_config_template(self, template_number):
         self.dry_run = is_true(get_default_config(config, "DryRun", False))
         self.meter_data = 0
         self.httptimeout = get_default_config(config, "HTTPTimeout", 2.5)
+        self._load_error_handling_config(config)
+
+    def _load_error_handling_config(self, config):
+        '''Loads error handling configuration values from the provided config object.'''
+
+        self.error_mode = get_default_config(config, "ErrorMode", constants.MODE_RETRYCOUNT).strip()
+        self.retry_after_seconds = int(get_default_config(config, "RetryAfterSeconds", 180))
+        self.min_retries_until_fail = int(get_default_config(config, "MinRetriesUntilFail", 3))
+        self.error_state_after_seconds = int(get_default_config(config, "ErrorStateAfterSeconds", 0))
 
     # get the Serialnumber
     def _get_serial(self, pvinverternumber):
@@ -539,61 +557,111 @@ def sign_of_life(self):
                      self.pvinverternumber, self._dbusservice["/Ac/Power"])
         return True
 
+    def _refresh_and_update(self):
+        """
+        Helper method to refresh data, handle data update if up-to-date, update index, and set successful flag.
+        """
+        self._refresh_data()
+        if self.is_data_up2date():
+            self._handle_data_update()
+        self._update_index()
+        return True
+
     def update(self):
         """
-        Updates the data from the DTU (Data Transfer Unit) and sets the DBus values if the data is up-to-date.
-
-        This method performs the following steps:
-        1. Refreshes the data from the DTU.
-        2. Checks if the data is up-to-date.
-        3. If in dry run mode, logs that no data is sent.
-        4. If not in dry run mode, sets the DBus values.
-        5. Updates the index.
-        6. Handles various exceptions that may occur during the update process:
-            - requests.exceptions.RequestException: Logs an HTTP error if the last update was successful.
-            - ValueError: Logs a value error if the last update was successful.
-            - Exception: Logs a general error if the last update was successful.
-        7. Logs a recovery message if the update was successful after a previous failure.
-
-        Attributes:
-            successful (bool): Indicates whether the update was successful.
+        Updates inverter data from the DTU (Data Transfer Unit) and sets DBus values if the data is up-to-date.
+
+        Main logic:
+        - In timeout mode: Always attempt reconnect every RetryAfterSeconds. Only set zero values after ErrorStateAfterSeconds has elapsed since last success.
+        - In retrycount mode: After min_retries_until_fail failures, wait RetryAfterSeconds before next attempt and set zero values immediately.
+        - Always updates the DBus update index after a refresh.
+        - Tracks success/failure state and manages reconnect timing.
+
+        Exception handling:
+        - Catches and logs HTTP, value, and general exceptions during update.
+        - Ensures update state is finalized regardless of outcome.
+
+        Returns:
+            None
         """
         logging.debug("_update")
         successful = False
+        now = time.time()
         try:
-            # update data from DTU once per _update call:
-            self._refresh_data()
-
-            if self.is_data_up2date():
-                if self.dry_run:
-                    logging.info("DRY RUN. No data is sent!!")
-                else:
-                    self.set_dbus_values()
-            self._update_index()
-            successful = True
+            if self.error_mode == constants.MODE_TIMEOUT and self.error_state_after_seconds > 0:
+                # Set zero values only after ErrorStateAfterSeconds has elapsed since last success
+                if (not self.last_update_successful and (now - self._last_update) >= self.error_state_after_seconds):
+                    self._handle_reconnect_wait()
+                # Always allow a reconnect attempt every RetryAfterSeconds
+                if (now - self._last_update) >= self.retry_after_seconds:
+                    successful = self._refresh_and_update()
+                # In normal operation (no error), always call _refresh_data on every update
+                if self.last_update_successful:
+                    successful = self._refresh_and_update()
+            elif self.error_mode == constants.MODE_RETRYCOUNT:
+                # Classic retry-count-based error handling
+                if self.failed_update_count >= self.min_retries_until_fail:
+                    self._handle_reconnect_wait()
+                # Determine if we should refresh data based on current state and timing
+                is_last_update_successful = self.last_update_successful
+                time_since_last_update = now - self._last_update
+                is_retry_interval_elapsed = time_since_last_update >= self.retry_after_seconds
+                is_below_min_retries = self.failed_update_count < self.min_retries_until_fail
+
+                should_refresh_data = (
+                    is_last_update_successful or
+                    is_retry_interval_elapsed or
+                    is_below_min_retries
+                )
+
+                if should_refresh_data:
+                    successful = self._refresh_and_update()
         except requests.exceptions.RequestException as exception:
-            if self.last_update_successful:
-                logging.warning(f"HTTP Error at _update for inverter "
-                                f"{self.pvinverternumber} ({self._get_name()}): {str(exception)}")
+            logging.warning(f"HTTP Error at _update for inverter "
+                            f"{self.pvinverternumber} ({self._get_name()}): {str(exception)}")
         except ValueError as error:
-            if self.last_update_successful:
-                logging.warning(f"Error at _update for inverter "
-                                f"{self.pvinverternumber} ({self._get_name()}): {str(error)}")
+            logging.warning(f"Error at _update for inverter "
+                            f"{self.pvinverternumber} ({self._get_name()}): {str(error)}")
         except Exception as error:  # pylint: disable=broad-except
-            if self.last_update_successful:
-                logging.warning(f"Error at _update for inverter "
-                                f"{self.pvinverternumber} ({self._get_name()})", exc_info=error)
+            logging.warning(f"Error at _update for inverter "
+                            f"{self.pvinverternumber} ({self._get_name()})", exc_info=error)
         finally:
-            if successful:
-                if not self.last_update_successful:
-                    logging.warning(
-                        f"Recovered inverter {self.pvinverternumber} ({self._get_name()}): "
-                        f"Successfully fetched data now: "
-                        f"{'NOT (yet?)' if not self.is_data_up2date() else 'Is'} up-to-date"
-                    )
-                    self.last_update_successful = True
-            else:
-                self.last_update_successful = False
+            self._finalize_update(successful)
+
+    def _handle_reconnect_wait(self):
+        if not self.reset_statuscode_on_next_success:
+            self.set_dbus_values_to_zero()
+            self.reset_statuscode_on_next_success = True
+
+    def _should_refresh_data(self, now):
+        return (
+            self.last_update_successful or
+            (now - self._last_update) >= self.retry_after_seconds or
+            self.failed_update_count < self.min_retries_until_fail
+        )
+
+    def _handle_data_update(self):
+        if self.dry_run:
+            logging.info("DRY RUN. No data is sent!!")
+        else:
+            self.set_dbus_values()
+
+    def _finalize_update(self, successful):
+        if successful:
+            if self.reset_statuscode_on_next_success:
+                self._dbusservice["/StatusCode"] = constants.STATUSCODE_RUNNING
+            if not self.last_update_successful:
+                logging.warning(
+                    f"Recovered inverter {self.pvinverternumber} ({self._get_name()}): "
+                    f"Successfully fetched data now: "
+                    f"{'NOT (yet?)' if not self.is_data_up2date() else 'Is'} up-to-date"
+                )
+            self.last_update_successful = True
+            self.failed_update_count = 0
+            self.reset_statuscode_on_next_success = False
+        else:
+            self.last_update_successful = False
+            self.failed_update_count += 1
 
     def _update_index(self):
         if self.dry_run:
@@ -657,12 +725,50 @@ def get_values_for_inverter(self):
 
         return (power, pvyield, current, voltage, dc_voltage)
 
+    def set_dbus_values_to_zero(self):
+        '''zero power data and cleat connection status and set dbus values'''
+
+        if self._servicename == "com.victronenergy.inverter":
+            # see https://github.com/victronenergy/venus/wiki/dbus#inverter
+            self._dbusservice["/Ac/Out/L1/V"] = 0
+            self._dbusservice["/Ac/Out/L1/I"] = 0
+            self._dbusservice["/Ac/Out/L1/P"] = 0
+            self._dbusservice["/Dc/0/Voltage"] = 0
+            self._dbusservice["/Ac/Power"] = 0
+
+            self._dbusservice["/Ac/L1/Current"] = 0
+            self._dbusservice["/Ac/L1/Power"] = 0
+            self._dbusservice["/Ac/L1/Voltage"] = 0
+        else:
+            # 0=Startup 0; 1=Startup 1; 2=Startup 2; 3=Startup 3; 4=Startup 4; 5=Startup 5; 6=Startup 6; 7=Running; 8=Standby; 9=Boot loading; 10=Error
+            self._dbusservice["/StatusCode"] = constants.STATUSCODE_ERROR
+
+            # three-phase inverter: split total power equally over all three phases
+            if "3P" == self.pvinverterphase:
+
+                self._dbusservice["/Ac/L1/Voltage"] = 0
+                self._dbusservice["/Ac/L1/Current"] = 0
+                self._dbusservice["/Ac/L1/Power"] = 0
+                self._dbusservice["/Ac/L2/Voltage"] = 0
+                self._dbusservice["/Ac/L2/Current"] = 0
+                self._dbusservice["/Ac/L2/Power"] = 0
+                self._dbusservice["/Ac/L3/Voltage"] = 0
+                self._dbusservice["/Ac/L3/Current"] = 0
+                self._dbusservice["/Ac/L3/Power"] = 0
+                self._dbusservice["/Ac/Power"] = 0
+
+            else:
+                pre = "/Ac/" + self.pvinverterphase
+                self._dbusservice[pre + "/Voltage"] = 0
+                self._dbusservice[pre + "/Current"] = 0
+                self._dbusservice[pre + "/Power"] = 0
+                self._dbusservice["/Ac/Power"] = 0
+
     def set_dbus_values(self):
         '''read data and set dbus values'''
         (power, pvyield, current, voltage, dc_voltage) = self.get_values_for_inverter()
         state = self.get_ac_inverter_state(current)
 
-        # This will be refactored later in classes
         if self._servicename == "com.victronenergy.inverter":
             # see https://github.com/victronenergy/venus/wiki/dbus#inverter
             self._dbusservice["/Ac/Out/L1/V"] = voltage