Merge branch 'main' into batch-poc

Author: jpcamara

Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    solid_queue (1.3.1)
+    solid_queue (1.4.0)
       activejob (>= 7.1)
       activerecord (>= 7.1)
       concurrent-ruby (>= 1.3.1)

README.md

@@ -17,6 +17,7 @@ Solid Queue can be used with SQL databases such as MySQL, PostgreSQL, or SQLite,
 - [Workers, dispatchers, and scheduler](#workers-dispatchers-and-scheduler)
   - [Fork vs. async mode](#fork-vs-async-mode)
 - [Configuration](#configuration)
+  - [Optional scheduler configuration](#optional-scheduler-configuration)
   - [Queue order and priorities](#queue-order-and-priorities)
   - [Queues specification and performance](#queues-specification-and-performance)
   - [Threads, processes, and signals](#threads-processes-and-signals)
@@ -32,6 +33,7 @@ Solid Queue can be used with SQL databases such as MySQL, PostgreSQL, or SQLite,
 - [Puma plugin](#puma-plugin)
 - [Jobs and transactional integrity](#jobs-and-transactional-integrity)
 - [Recurring tasks](#recurring-tasks)
+  - [Scheduling and unscheduling recurring tasks dynamically](#scheduling-and-unscheduling-recurring-tasks-dynamically)
 - [Inspiration](#inspiration)
 - [License](#license)
 
@@ -210,7 +212,7 @@ By default, Solid Queue will try to find your configuration under `config/queue.
 bin/jobs -c config/calendar.yml
 ```
 
-You can also skip all recurring tasks by setting the environment variable `SOLID_QUEUE_SKIP_RECURRING=true`. This is useful for environments like staging, review apps, or development where you don't want any recurring jobs to run. This is equivalent to using the `--skip-recurring` option with `bin/jobs`.
+You can also skip the scheduler process by setting the environment variable `SOLID_QUEUE_SKIP_RECURRING=true`. This is useful for environments like staging, review apps, or development where you don't want any recurring jobs to run. This is equivalent to using the `--skip-recurring` option with `bin/jobs`.
 
 This is what this configuration looks like:
 
@@ -228,6 +230,10 @@ production:
       threads: 5
       polling_interval: 0.1
       processes: 3
+  scheduler:
+    dynamic_tasks_enabled: true
+    polling_interval: 5
+
 ```
 
 Everything is optional. If no configuration at all is provided, Solid Queue will run with one dispatcher and one worker with default settings. If you want to run only dispatchers or workers, you just need to include that section alone in the configuration. For example, with the following configuration:
@@ -272,6 +278,19 @@ It is recommended to set this value less than or equal to the queue database's c
 - `concurrency_maintenance`: whether the dispatcher will perform the concurrency maintenance work. This is `true` by default, and it's useful if you don't use any [concurrency controls](#concurrency-controls) and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else.
 
 
+### Optional scheduler configuration
+
+Optionally, you can configure the scheduler process under the `scheduler` section in your `config/queue.yml` if you'd like to [schedule recurring tasks dynamically](#scheduling-and-unscheduling-recurring-tasks-dynamically).
+
+```yaml
+scheduler:
+  dynamic_tasks_enabled: true
+  polling_interval: 5
+```
+
+- `dynamic_tasks_enabled`: whether the scheduler should poll for [dynamically scheduled recurring tasks](#scheduling-and-unscheduling-recurring-tasks-dynamically). This is `false` by default. When enabled, the scheduler will poll the database at the given `polling_interval` to pick up tasks scheduled via `SolidQueue.schedule_recurring_task`.
+- `polling_interval`: how frequently (in seconds) the scheduler checks for dynamic task changes. Defaults to `5`.
+
 ### Queue order and priorities
 
 As mentioned above, if you specify a list of queues for a worker, these will be polled in the order given, such as for the list `real_time,background`, no jobs will be taken from `background` unless there aren't any more jobs waiting in `real_time`.
@@ -463,7 +482,7 @@ class MyJob < ApplicationJob
 - `group` is used to control the concurrency of different job classes together. It defaults to the job class name.
 - `on_conflict` controls behaviour when enqueuing a job that conflicts with the concurrency limits configured. It can be set to one of the following:
   - (default) `:block`: the job is blocked and is dispatched when another job completes and unblocks it, or when the duration expires.
-  - `:discard`: the job is discarded. When you choose this option, bear in mind that if a job runs and fails to remove the concurrency lock (or _semaphore_, read below to know more about this), all jobs conflicting with it will be discarded up to the interval defined by `duration` has elapsed.
+  - `:discard`: the job is discarded. When you choose this option, bear in mind that if a job runs and fails to remove the concurrency lock (or _semaphore_, read below to know more about this), all jobs conflicting with it will be discarded until the interval defined by `duration` has elapsed.
 
 When a job includes these controls, we'll ensure that, at most, the number of jobs (indicated as `to`) that yield the same `key` will be performed concurrently, and this guarantee will last for `duration` for each job enqueued. Note that there's no guarantee about _the order of execution_, only about jobs being performed at the same time (overlapping).
 
@@ -473,7 +492,7 @@ Since something can happen that prevents the first job from releasing the semaph
 
 It's important to note that after one or more candidate jobs are unblocked (either because a job finishes or because `duration` expires and a semaphore is released), the `duration` timer for the still blocked jobs is reset. This happens indirectly via the expiration time of the semaphore, which is updated.
 
-When using `discard` as the behaviour to handle conflicts, you might have jobs discarded for up to the `duration` interval if something happens and a running job fails to release the semaphore.
+When using `discard` as the behaviour to handle conflicts, you might have jobs discarded for until the `duration` interval if something happens and a running job fails to release the semaphore.
 
 
 For example:
@@ -793,6 +812,38 @@ my_periodic_resque_job:
 
 and the job will be enqueued via `perform_later` so it'll run in Resque. However, in this case we won't track any `solid_queue_recurring_execution` record for it and there won't be any guarantees that the job is enqueued only once each time.
 
+### Scheduling and unscheduling recurring tasks dynamically
+
+You can schedule and unschedule recurring tasks at runtime, without editing the configuration file. To enable this, you need to set `dynamic_tasks_enabled: true` in the `scheduler` section of your `config/queue.yml`, [as explained earlier](#optional-scheduler-configuration).
+
+```yaml
+scheduler:
+  dynamic_tasks_enabled: true
+```
+
+Then you can use the following methods to add recurring tasks dynamically:
+
+```ruby
+SolidQueue.schedule_recurring_task(
+  "my_dynamic_task",
+  class: "MyJob",
+  args: [1, 2],
+  schedule: "every 10 minutes"
+)
+```
+
+This accepts the same options as the YAML configuration: `class`, `args`, `command`, `schedule`, `queue`, `priority`, and `description`.
+
+To remove a dynamically scheduled task:
+
+```ruby
+SolidQueue.unschedule_recurring_task("my_dynamic_task")
+```
+
+Only dynamic tasks can be unscheduled at runtime. Attempting to unschedule a static task (defined in `config/recurring.yml`) will raise an `ActiveRecord::RecordNotFound` error.
+
+Tasks scheduled like this persist between Solid Queue's restarts and won't stop running until you manually unschedule them. 
+
 ## Inspiration
 
 Solid Queue has been inspired by [resque](https://github.com/resque/resque) and [GoodJob](https://github.com/bensheldon/good_job). We recommend checking out these projects as they're great examples from which we've learnt a lot.

app/models/solid_queue/blocked_execution.rb

@@ -26,7 +26,9 @@ def release_many(concurrency_keys)
 
       def release_one(concurrency_key)
         transaction do
-          if execution = ordered.where(concurrency_key: concurrency_key).limit(1).non_blocking_lock.first
+          if execution = ordered.where(concurrency_key: concurrency_key).limit(1)
+              .use_index(:index_solid_queue_blocked_executions_for_release)
+              .non_blocking_lock.first
             execution.release
           end
         end

app/models/solid_queue/failed_execution.rb

@@ -6,7 +6,7 @@ class FailedExecution < Execution
 
     serialize :error, coder: JSON
 
-    before_create :expand_error_details_from_exception
+    before_save :expand_error_details_from_exception, if: :exception
 
     attr_accessor :exception
 

app/models/solid_queue/job/concurrency_controls.rb

@@ -26,7 +26,7 @@ def unblock_next_blocked_job
       end
 
       def concurrency_limited?
-        concurrency_key.present?
+        concurrency_key.present? && job_class.present?
       end
 
       def blocked?

app/models/solid_queue/job/retryable.rb

@@ -16,7 +16,16 @@ def retry
       end
 
       def failed_with(exception)
-        FailedExecution.create_or_find_by!(job_id: id, exception: exception)
+        FailedExecution.transaction(requires_new: true) do
+          FailedExecution.create!(job_id: id, exception: exception)
+        end
+      rescue ActiveRecord::RecordNotUnique
+        if (failed_execution = FailedExecution.find_by(job_id: id))
+          failed_execution.exception = exception
+          failed_execution.save!
+        else
+          retry
+        end
       end
 
       def reset_execution_counters

app/models/solid_queue/ready_execution.rb

@@ -30,7 +30,8 @@ def select_and_lock(queue_relation, process_id, limit)
         end
 
         def select_candidates(queue_relation, limit)
-          queue_relation.ordered.limit(limit).non_blocking_lock.select(:id, :job_id)
+          # Force query execution here with #to_a to avoid unintended FOR UPDATE query executions
+          queue_relation.ordered.limit(limit).non_blocking_lock.select(:id, :job_id).to_a
         end
 
         def lock_candidates(executions, process_id)

app/models/solid_queue/record.rb

@@ -20,6 +20,13 @@ def supports_insert_conflict_target?
           connection.supports_insert_conflict_target?
         end
       end
+
+      # Pass index hints to the query optimizer using SQL comment hints.
+      # Uses MySQL 8 optimizer hint query comments, which SQLite and
+      # PostgreSQL ignore.
+      def use_index(*indexes)
+        optimizer_hints "INDEX(#{quoted_table_name} #{indexes.join(', ')})"
+      end
     end
   end
 end

app/models/solid_queue/recurring_task.rb

@@ -6,11 +6,12 @@ module SolidQueue
   class RecurringTask < Record
     serialize :arguments, coder: Arguments, default: []
 
-    validate :supported_schedule
+    validate :ensure_schedule_supported
     validate :ensure_command_or_class_present
-    validate :existing_job_class
+    validate :ensure_existing_job_class
 
     scope :static, -> { where(static: true) }
+    scope :dynamic, -> { where(static: false) }
 
     has_many :recurring_executions, foreign_key: :task_key, primary_key: :key
 
@@ -32,7 +33,15 @@ def from_configuration(key, **options)
           queue_name: options[:queue].presence,
           priority: options[:priority].presence,
           description: options[:description],
-          static: true
+          static: options.fetch(:static, true)
+      end
+
+      def create_dynamic_task(key, **options)
+        from_configuration(key, **options.merge(static: false)).save!
+      end
+
+      def delete_dynamic_task(key)
+        RecurringTask.dynamic.find_by!(key: key).destroy
       end
 
       def create_or_update_all(tasks)
@@ -102,10 +111,19 @@ def attributes_for_upsert
     end
 
     private
-      def supported_schedule
+      def ensure_schedule_supported
         unless parsed_schedule.instance_of?(Fugit::Cron)
           errors.add :schedule, :unsupported, message: "is not a supported recurring schedule"
         end
+      rescue ArgumentError => error
+        message = if error.message.include?("multiple crons")
+          "generates multiple cron schedules. Please use separate recurring tasks for each schedule, " +
+          "or use explicit cron syntax (e.g., '40 0,15 * * *' for multiple times with the same minutes)"
+        else
+          error.message
+        end
+
+        errors.add :schedule, :unsupported, message: message
       end
 
       def ensure_command_or_class_present
@@ -114,7 +132,7 @@ def ensure_command_or_class_present
         end
       end
 
-      def existing_job_class
+      def ensure_existing_job_class
         if class_name.present? && job_class.nil?
           errors.add :class_name, :undefined, message: "doesn't correspond to an existing class"
         end
@@ -152,7 +170,7 @@ def arguments_with_kwargs
 
 
       def parsed_schedule
-        @parsed_schedule ||= Fugit.parse(schedule)
+        @parsed_schedule ||= Fugit.parse(schedule, multi: :fail)
       end
 
       def job_class

app/models/solid_queue/semaphore.rb

@@ -40,7 +40,7 @@ def initialize(job)
       end
 
       def wait
-        if semaphore = Semaphore.find_by(key: key)
+        if semaphore = Semaphore.lock.find_by(key: key)
           semaphore.value > 0 && attempt_decrement
         else
           attempt_creation