From 01b8b57ef453bbc8d4561ed4c7aa0f7c0abdf9e4 Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Fri, 26 Sep 2025 21:11:35 +0000
Subject: [PATCH 1/3] Initial plan
From 1bf327487dac60616be8c4c1780c62de047dbcdb Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Fri, 26 Sep 2025 21:14:45 +0000
Subject: [PATCH 2/3] Initial plan for setting up Copilot instructions
Co-authored-by: masilver99 <14352629+masilver99@users.noreply.github.com>
---
.../AgQueue/AgQueue.Common/AgQueue.Common.xml | 279 ++++++++++
.../AgQueue.Postgres/AgQueue.Postgres.xml | 137 +++++
.../AgQueue.Library/AgQueue.Library.xml | 503 ++++++++++++++++++
.../AgQueue/AgQueue.Sqlite/AgQueue.Sqlite.xml | 137 +++++
4 files changed, 1056 insertions(+)
create mode 100644 AgQueue.Common/C:/dev/AgQueue/AgQueue.Common/AgQueue.Common.xml
create mode 100644 AgQueue.Postgres/C:/dev/AgQueue/AgQueue.Postgres/AgQueue.Postgres.xml
create mode 100644 AgQueue.Server.Common/C:/dev/AgQueue/AgQueue.Library/AgQueue.Library.xml
create mode 100644 AgQueue.Sqlite/C:/dev/AgQueue/AgQueue.Sqlite/AgQueue.Sqlite.xml
diff --git a/AgQueue.Common/C:/dev/AgQueue/AgQueue.Common/AgQueue.Common.xml b/AgQueue.Common/C:/dev/AgQueue/AgQueue.Common/AgQueue.Common.xml
new file mode 100644
index 0000000..0c29af7
--- /dev/null
+++ b/AgQueue.Common/C:/dev/AgQueue/AgQueue.Common/AgQueue.Common.xml
@@ -0,0 +1,279 @@
+
+
+
+ AgQueue.Common
+
+
+
+
+ Extensions methods for use on Objects.
+
+
+
+
+ Throws ArgumentNullException if object is null.
+
+ Object to perform null check on.
+ Param name to include in exception message.
+
+
+
+ Extensions methods to be used on stings.
+
+
+
+
+ Standardizes the queue name so it's always consistent.
+
+ Raw queue name.
+ Standardized queue name.
+
+
+
+ Interface for Client to implement.
+
+
+
+
+ Current state of a message.
+
+
+
+
+ Gets active value. Is the message is live and can be pulled from the queue.
+
+
+
+
+ Gets InTransaction. Means the message is currently tied to a transaction, either during insert of during processing. IOW, this message is currently being inserted or pulled from the queue.
+
+
+
+
+ Gets processed state. i.e. This message has been processed and will not be pulled.
+
+
+
+
+ Gets if Message has expired and will not be pulled.
+
+
+
+
+ Gets boolean representing if RetryExceeded. Message retry limit has been reached and message will no longer be pulled.
+
+
+
+
+ Represents a Queue Message.
+
+
+
+
+ Gets generated unique message id.
+
+
+
+
+ Gets the id of the queue.
+
+
+
+
+ Gets the id of the queue transaction.
+
+
+
+
+ Gets the transaction action. i.e. was this message added or pulled in the transaction.
+
+
+
+
+ Gets the datetime the message was added.
+
+
+
+
+ Gets the datetime the message was closed, i.e. processed or cancelled or expired.
+
+
+
+
+ Gets the priority of the message. Lower is higher priority.
+
+
+
+
+ Gets number of attempts to have message processed, i.e. commited.
+
+
+
+
+ Gets the number of rollbacks or timeouts before the message expires.
+
+
+
+
+ Gets DateTime the message will expire.
+
+
+
+
+ Gets the interger correlation id, used by calling application.
+
+
+
+
+ Gets string group name. Used by external application for grouping purposes.
+
+
+
+
+ Gets actual message data.
+
+
+
+
+ Gets Message State, i.e. Active, closed, etc.
+
+
+
+
+ Gets string based metadata describing the message in more detail. Optional.
+
+
+
+
+ Represents a Queue.
+
+
+
+
+ Gets the unique queue id.
+
+
+
+
+ Gets the name of the queue.
+
+
+
+
+ Represents a Queue Transaction.
+
+
+
+
+ Gets the unique ID for a transaction.
+
+
+
+
+ Gets a value indicating the state of the transaction.
+
+
+
+
+ Gets the date and time the transaction was created.
+
+
+
+
+ Gets the date and time the transaction will expire. e.g. after this datetime, the transaction will automatically rollback.
+
+
+
+
+ Gets the date and time the transaction was closed, null if not closed.
+
+
+
+
+ Represents the action performed within a transaction.
+
+
+
+
+ This should never happen.
+
+
+
+
+ Item was added within a transaction (if rolledback, delete).
+
+
+
+
+ Item was pulled within transaction, increment retry count if rolledback.
+
+
+
+
+ Result of transaction commital.
+
+ Only commits use the results since commits may have valid responses. If a rollback fails, an exception should be thrown, at least for now.
+
+
+
+ Transaction was successfully committed.
+
+
+
+
+ Transaction expired before the commit could be completed.
+
+
+
+
+ The transaction was already closed.
+
+
+
+
+ Transaction could not be found. Perhaps this should be an exception.
+
+
+
+
+ The current state of a transaction.
+
+
+
+
+ This should never occur. It means there is a serious bug.
+
+
+
+
+ Transaction is active.
+
+
+
+
+ Transaction has been committed by user.
+
+
+
+
+ Transaction was rolled back by user.
+
+
+
+
+ Transaction was automatically expired due to timeout.
+
+
+
+
+ Exception thrown for unique WorkQueue exceptions.
+
+
+
+
+ Initializes a new instance of the class.
+
+ Error message.
+
+
+
diff --git a/AgQueue.Postgres/C:/dev/AgQueue/AgQueue.Postgres/AgQueue.Postgres.xml b/AgQueue.Postgres/C:/dev/AgQueue/AgQueue.Postgres/AgQueue.Postgres.xml
new file mode 100644
index 0000000..311c700
--- /dev/null
+++ b/AgQueue.Postgres/C:/dev/AgQueue/AgQueue.Postgres/AgQueue.Postgres.xml
@@ -0,0 +1,137 @@
+
+
+
+ AgQueue.Postgres
+
+
+
+
+ Wrapper around SQLite transaction. Used by storage classes.
+
+
+
+
+ Initializes a new instance of the class.
+
+ SQLite connection.
+
+
+
+ Gets the internal SQLite transaction.
+
+
+
+
+
+
+
+
+
+
+ Extensions on the IStorageTransaction interface.
+
+
+
+
+ Converts the IStorageTransaction to a NpgsqlTransaction.
+
+ The IStorageTransaction to convert.
+ SqliteTransaction.
+
+
+
+ Implements the IStorage interface for storing and retrieving queue date to SQLite.
+
+
+
+
+ Initializes a new instance of the class.
+
+ The Sqlite connection string.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Executes an anonymous method wrapped with robust logging, command line options loading, and error handling.
+
+ Options class to load from command line.
+ The anonymous method execute. Contains a logging object and the options object, both of which can be accessed in the anonymous method.
+ Sqlite connection.
+ Returns generic object T.
+
+
+
+ Executes an anonymous method wrapped with robust logging, command line options loading, and error handling.
+
+ Options class to load from command line.
+ The anonymous method execute. Contains a logging object and the options object, both of which can be accessed in the anonymous method.
+ Sqlite connection.
+ Returns generic object T.
+
+
+
diff --git a/AgQueue.Server.Common/C:/dev/AgQueue/AgQueue.Library/AgQueue.Library.xml b/AgQueue.Server.Common/C:/dev/AgQueue/AgQueue.Library/AgQueue.Library.xml
new file mode 100644
index 0000000..97022fb
--- /dev/null
+++ b/AgQueue.Server.Common/C:/dev/AgQueue/AgQueue.Library/AgQueue.Library.xml
@@ -0,0 +1,503 @@
+
+
+
+ AgQueue.Server.Common
+
+
+
+
+ Starting point for accessing all queue related APIS
+ This is mostly a factory for creating Queues and Transactions.
+
+
+ Exceptions are not used unless there is an exceptional condition. For example, if an items doesn't exist or a param is invalid,
+ this is handled without an exception. This is mostly for speed and simplicity with the gRPC interface.
+
+
+
+
+ Initializes a new instance of the class.
+
+ The storage implementation to use for storage of queues and messages.
+ The QueueOptions object passed in via DI.
+
+
+
+ Creates a new queue.
+
+ The name of the queue.
+ A Queue info object. Note: the queueName is the stadnardized name.
+
+
+
+ Delete a queue and all messages in the queue.
+ Throws an exception if Queue doesn't exist.
+
+ Name of the queue to delete.
+
+
+
+ Returns information about the requested queue.
+
+ The ID of the queue.
+ QueueInfo object. Returns null if not found.
+
+
+
+ Returns information about the requested queue.
+
+ The Name of the queue to lookup.
+ QueueInfo object. Null if not found.
+
+
+
+ Deletes a queue and 1) rollsback any transaction related to the queue, 2) deletes all messages in the queue.
+
+ Queue id.
+ ValueTask.
+
+
+
+ Disposes of storage resources.
+
+
+
+
+ Creates underlying structure in the storage layer.
+
+ Will delete and recreate underlying structures.
+ Returns ValueTask.
+
+
+
+ Starts a transaction used my add message and pull message.
+
+ Override default expiration time.
+ Transaction ID.
+
+
+
+ Extends the transaction by x number of minutes (from the current datetime).
+
+ The transaction to extend.
+ How long to extend the transaction by.
+ ValueTask.
+
+
+
+ View the next message in the spcified queue.
+
+ Keep in mind, this may change by the time of the dequeue call.
+ The queue to peek from.
+ Message object or null if no message available.
+
+
+
+ Retrieve a message by it's message ID.
+
+ Keep in mind, this may change by the time of the dequeue call.
+ The ID of the message to retrieve.
+ Message object or null if no message available.
+
+
+
+ Commits Transaction, updating all messages in transaction.
+
+ Transaction Id to commit.
+ ValueTask.
+
+
+
+ Rollsback a transaction, undoing any changes to messages in the transaction.
+
+ The transaction ID of the transaction to rollback.
+ ValueTask.
+
+
+
+ Adds a message to the specified queue.
+
+ The transaction to add the message in.
+ The queue the message will be placed in.
+ The message payload.
+ Message metadata.
+ The message priority. Higher numbers will have higher priority.
+ How many times should the message pull fail before being retired.
+ How long in minutes before the message will expire before it's pulled. 0 = no expiration.
+ Correlation number.
+ Group name applied to message.
+ Return the message id.
+
+
+
+ Dequeues a message from a specific queue.
+
+ The transaction ID of the transaction the message will be dequeued in.
+ The queue to pull the messge from.
+ Returns a message object or null if there is no message to dequeue.
+
+
+
+ Expires transactions, message and checks message counts, maybe.
+
+ The current time is passed in so it is consistent with the time used in the calling procedure.
+ ValueTask.
+
+
+
+ The interface for storing and retrieving queue information from a storage mechinism, usually a database.
+ When implementing IStorage, use the StorageSqlite as an example. There should be no business logic in
+ classes that implement IStorage.
+
+
+
+
+ Called when Queue process starts. Connections to the storage should be made here, etc.
+
+ Should all existing queues and messages be deleted.
+ ValueTask.
+
+
+
+ Create a new Queue in storage.
+
+ Queue name.
+ ValueTask.
+
+
+
+ Delete a Queue and ALL messages in the Queue.
+
+ Queue Id of the queue to delete.
+ ValueTask.
+
+
+
+ Returns a Transaction object or null value if not found.
+
+ The Id of the transaction to lookup.
+ Transaction object or null if not found.
+
+
+
+ Add a message to the storage.
+
+ Queue Transaction ID.
+ ID of the storage queue.
+ Message data.
+ Datetime the message was added.
+ String metadata on message data.
+ Message priority.
+ How many retries before expires.
+ Datetime the message will expire.
+ Correlation ID.
+ Group name.
+ Always will be "add".
+ "Always will be in transaction.
+ Message ID.
+
+
+
+ Returns the id and name of the Queue. If no queue is found, returns null.
+
+
+ This search should be case sensitive, only use LIKE with SQLite.
+
+ Name of the queue to lookup.
+ QueueInfo containing ID and Name of queue. Null if not found.
+
+
+
+ Returns the id and name of the Queue. If no queue is found, returns null.
+
+
+ This search should be case sensitive, only use LIKE with SQLite.
+
+ ID of the queue to lookup.
+ QueueInfo containing ID and Name of queue. Null if not found.
+
+
+
+ Starts a transaction for use when adding or pulling messages.
+
+ When the transaction started.
+ When the transaction will end.
+ Returns transaction ID.
+
+
+
+ Updates a set of messages.
+
+ The storage transaction to make change within.
+ The transaction the messages are associated with.
+ The TransactionAction to search for.
+ Current message state.
+ Update the message state to this value.
+ Returns the number of messages updated.
+
+
+
+ Updates the message retry cuont based on transactionAction and MessageState.
+
+ The storage transaction to run the query under.
+ The messages must be in this transaction.
+ The transactionAction the message must be in.
+ The messageState the message must be in.
+ Number of messages updated.
+
+
+
+ Delete messages that were added in the specified transtacion.
+
+ The storage transaction to run the query in.
+ The transaction the messages must be in.
+ Number of records deleted.
+
+
+
+ Extends the transaction's expiration datetime.
+
+ The id of the transaction to update.
+ The new expiration datetime.
+ ValueTask.
+
+
+
+ Update the transaction's state and end datetime.
+
+ The storage transaction to make change within.
+ The id of the transaction to update.
+ The new state of the transaction.
+ Reason the transaction was ended. Optional.
+ Datetime the transaction was closed (or null if not closed).
+ ValueTask.
+
+
+
+ Starts a storage (database) transaction, not a queue transaction.
+
+
+ Not all Storage classes will have internal transactions, so this can return a dummy class that performs no actions.
+
+ Returns a class represented by IStorageTransaction which can commit or rollbacl the transaction.
+
+
+
+ Deletes added messages in an expired transaction.
+
+ Storage Transaction to run under.
+ DateTime to expire against.
+ Returns the count of the deleted records.
+
+
+
+ Updates the retry counts for messages in an expired transaction.
+
+ Storage Transaction to run under.
+ DateTime to expire against.
+ Number of records updated.
+
+
+
+ Expires transactions whose expiry date time is past the currentDateTime.
+
+ Storage Transaction to run under.
+ DateTime to expire against.
+ Returns the number of transactions expired.
+
+
+
+ Expire messages that are past their expiration date.
+
+ Date used to validate expiration date against. Also to be used as the close datetime.
+ Number of messages updated.
+
+
+
+ Close message that are beyond their retry count.
+
+ DateTime to use a close datetime.
+ Number of messages closed.
+
+
+
+ Dequeue the next message in the queue and flag the message with the transaction id.
+
+ The transaction id to add to the message.
+ The queue to pull the message frrom.
+ Message object or null of no message to pull.
+
+
+
+ View the next message in the queue, without removing it from queue.
+
+ The queue to pull the message from.
+ Message object or null of no message to pull.
+
+
+
+ View a message by the message ID.
+
+ The ID of the message to view.
+ Message object or null of no message to view.
+
+
+
+ Represents a transaction used by the storage (usually a database transaction).
+
+
+
+
+ Commits the transaction, usually this reprents a database transaction.
+
+
+
+
+ Rollsback the transaction, usually this reprents a database transaction.
+
+
+
+
+ Contains information a specific queue.
+
+
+
+
+ Gets or sets queue name.
+
+
+
+
+ Gets or sets queue ID.
+
+
+
+
+ Options used by the queue server.
+
+
+
+
+ How many minutes until a message expires. Will be overriden by the value in the message, unless it's zero.
+ 0 = max message timeout. 0 in the message, means use the detault here..
+
+
+
+
+ How long until a transaction expires in minutes. The value in the transaction will override this value,
+ unless it zero. If this is zero as well, transactions won't expire.
+
+
+
+
+ Result codes from internal API calls. This currently map directly to gRPC status codes, but
+ the internal api may be used with another protocol, hence it can't depend on gRPC status codes.
+ Most of these are not used by the Internal API.
+
+
+
+
+ Not an error; returned on success.
+
+
+
+
+ The operation was cancelled (typically by the caller).
+
+
+
+
+ Unknown error. An example of where this error may be returned is if a Status
+ value received from another address space belongs to an error-space that is not
+ known in this address space. Also errors raised by APIs that do not return enough
+ error information may be converted to this error.
+
+
+
+
+ Client specified an invalid argument. Note that this differs from FAILED_PRECONDITION.
+ INVALID_ARGUMENT indicates arguments that are problematic regardless of the state
+ of the system (e.g., a malformed file name).
+
+
+
+
+ Deadline expired before operation could complete. For operations that change
+ the state of the system, this error may be returned even if the operation has
+ completed successfully. For example, a successful response from a server could
+ have been delayed long enough for the deadline to expire.
+
+
+
+
+ Some requested entity (e.g., file or directory) was not found.
+
+
+
+
+ Some entity that we attempted to create (e.g., file or directory) already exists.
+
+
+
+
+ The caller does not have permission to execute the specified operation. PERMISSION_DENIED
+ must not be used for rejections caused by exhausting some resource (use RESOURCE_EXHAUSTED
+ instead for those errors). PERMISSION_DENIED must not be used if the caller can
+ not be identified (use UNAUTHENTICATED instead for those errors).
+
+
+
+
+ Some resource has been exhausted, perhaps a per-user quota, or perhaps the entire
+ file system is out of space.
+
+
+
+
+ Operation was rejected because the system is not in a state required for the
+ operation's execution. For example, directory to be deleted may be non-empty,
+ an rmdir operation is applied to a non-directory, etc.
+
+
+
+
+ The operation was aborted, typically due to a concurrency issue like sequencer
+ check failures, transaction aborts, etc.
+
+
+
+
+ Operation was attempted past the valid range. E.g., seeking or reading past end
+ of file.
+
+
+
+
+ Operation is not implemented or not supported/enabled in this service.
+
+
+
+
+ Internal errors. Means some invariants expected by underlying system has been
+ broken. If you see one of these errors, something is very broken.
+
+
+
+
+ The service is currently unavailable. This is a most likely a transient condition
+ and may be corrected by retrying with a backoff. Note that it is not always safe
+ to retry non-idempotent operations.
+
+
+
+
+ Unrecoverable data loss or corruption.
+
+
+
+
+ The request does not have valid authentication credentials for the operation.
+
+
+
+
diff --git a/AgQueue.Sqlite/C:/dev/AgQueue/AgQueue.Sqlite/AgQueue.Sqlite.xml b/AgQueue.Sqlite/C:/dev/AgQueue/AgQueue.Sqlite/AgQueue.Sqlite.xml
new file mode 100644
index 0000000..c401fab
--- /dev/null
+++ b/AgQueue.Sqlite/C:/dev/AgQueue/AgQueue.Sqlite/AgQueue.Sqlite.xml
@@ -0,0 +1,137 @@
+
+
+
+ AgQueue.Sqlite
+
+
+
+
+ Wrapper around SQLite transaction. Used by storage classes.
+
+
+
+
+ Initializes a new instance of the class.
+
+ SQLite connection.
+
+
+
+ Gets the internal SQLite transaction.
+
+
+
+
+
+
+
+
+
+
+ Extensions on the IStorageTransaction interface.
+
+
+
+
+ Converts the IStorageTransaction to a SqliteTransaction.
+
+ The IStorageTransaction to convert.
+ SqliteTransaction.
+
+
+
+ Implements the IStorage interface for storing and retriving queue date to SQLite.
+
+
+
+
+ Initializes a new instance of the class.
+
+ The Sqlite connection string.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Executes an anonymous method wrapped with robust logging, command line options loading, and error handling.
+
+ Options class to load from command line.
+ The anonymous method execute. Contains a logging object and the options object, both of which can be accessed in the anonymous method.
+ Sqlite connection.
+ Returns generic object T.
+
+
+
+ Executes an anonymous method wrapped with robust logging, command line options loading, and error handling.
+
+ Options class to load from command line.
+ The anonymous method execute. Contains a logging object and the options object, both of which can be accessed in the anonymous method.
+ Sqlite connection.
+ Returns generic object T.
+
+
+
From 6f00d22f9a67405f3c8087c1e56d882a5e5fe21c Mon Sep 17 00:00:00 2001
From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com>
Date: Fri, 26 Sep 2025 21:17:51 +0000
Subject: [PATCH 3/3] Add comprehensive GitHub Copilot instructions
Co-authored-by: masilver99 <14352629+masilver99@users.noreply.github.com>
---
.github/copilot-instructions.md | 163 ++++++
.../AgQueue/AgQueue.Common/AgQueue.Common.xml | 279 ----------
.../AgQueue.Postgres/AgQueue.Postgres.xml | 137 -----
.../AgQueue.Library/AgQueue.Library.xml | 503 ------------------
.../AgQueue/AgQueue.Sqlite/AgQueue.Sqlite.xml | 137 -----
5 files changed, 163 insertions(+), 1056 deletions(-)
create mode 100644 .github/copilot-instructions.md
delete mode 100644 AgQueue.Common/C:/dev/AgQueue/AgQueue.Common/AgQueue.Common.xml
delete mode 100644 AgQueue.Postgres/C:/dev/AgQueue/AgQueue.Postgres/AgQueue.Postgres.xml
delete mode 100644 AgQueue.Server.Common/C:/dev/AgQueue/AgQueue.Library/AgQueue.Library.xml
delete mode 100644 AgQueue.Sqlite/C:/dev/AgQueue/AgQueue.Sqlite/AgQueue.Sqlite.xml
diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md
new file mode 100644
index 0000000..f63072b
--- /dev/null
+++ b/.github/copilot-instructions.md
@@ -0,0 +1,163 @@
+# GitHub Copilot Instructions for AgQueue
+
+## Project Overview
+
+AgQueue is a simple, opinionated, transactional work queue system built in C# with .NET 5. It transforms any supported database (currently SQLite and PostgreSQL) into a full-fledged work queue with advanced messaging capabilities.
+
+### Key Features
+- **Transactional**: Transactions are mandatory with expiry timespans and can be extended
+- **Priority Support**: Messages returned by priority, then FIFO
+- **Durable Storage**: Messages reliably stored to disk using SQLite/PostgreSQL
+- **Multiple Queues**: Support for multiple named queues
+- **Built-in Retry**: Automatic message retry with configurable delays
+- **gRPC Communication**: Primary communication via gRPC server
+- **Message Metadata**: Custom data storage for messages with tags and correlation IDs
+
+## Architecture
+
+### Project Structure
+- **AgQueue.Server.Common**: Core internal API and storage interfaces
+- **AgQueue.Common**: Client interfaces and common models
+- **AgQueue.GrpcServer**: gRPC server implementation
+- **AgQueue.Sqlite**: SQLite storage implementation
+- **AgQueue.Postgres**: PostgreSQL storage implementation
+- **AgQueue.Models**: Shared data models and protobuf definitions
+- **Benchmarks**: Performance benchmarking project
+- **tests/**: Integration and unit tests
+
+### Key Components
+1. **InternalApi**: Core business logic for queue operations
+2. **IStorage/IStorageTransaction**: Storage abstraction layer
+3. **AgQueueService**: gRPC service implementation
+4. **Transaction Management**: Two-level transaction system (queue + database)
+5. **protobuf/gRPC**: Protocol definitions in `AgQueue.Models/AgQueue.proto`
+
+### gRPC API Structure
+The main service `QueueApi` provides these operations:
+- **Queue Management**: CreateQueue, DeleteQueue, GetQueueInfo
+- **Transaction Control**: StartTransaction, CommitTransaction, RollbackTransaction
+- **Message Operations**: QueueMessage, DequeueMessage, PeekMessage
+- **Storage Management**: InitializeStorage
+
+## Development Guidelines
+
+### Code Standards
+- Follow StyleCop Analyzers recommendations (configured in stylecop.json)
+- **Zero warnings policy**: Pull requests must have no compiler warnings
+- Use XML documentation (`///`) for all public methods and properties
+- Company name: "Michael Silver" (configured in stylecop.json)
+- Copyright: "Copyright (c) Michael Silver. All rights reserved."
+
+### Coding Conventions
+- Use int64 for primary keys (application-generated, not database-generated)
+- Queue names are case sensitive
+- Prefer explicit error handling over exceptions where appropriate
+- Use `ResultCode` enum for internal API responses (maps to gRPC status codes)
+
+### Key Patterns
+
+#### Transaction Handling
+```csharp
+// Two types of transactions to be aware of:
+// 1. Database transactions (IStorageTransaction) - for SQL atomicity
+// 2. Queue transactions (stored in Transaction table) - for message handling
+```
+
+#### Storage Implementation
+- Implement `IStorage` interface for new database providers
+- Reference `StorageSqlite` class as the primary example
+- Handle connection management and transaction lifecycle properly
+
+#### Error Handling
+- Use `ApiResult` for internal API returns
+- Map `ResultCode` to appropriate gRPC status codes
+- Implement proper exception interceptors for gRPC calls
+
+### Build and Testing
+- Target Framework: .NET 5.0 (note: end-of-life, may require upgrade to newer .NET version)
+- Build with: `dotnet build --configuration Release`
+- Run tests: `dotnet test` (requires .NET 5.0 runtime)
+- No warnings should be present in successful builds
+- Integration tests use in-process gRPC server on port 10043
+- Current build shows package vulnerabilities (MessagePack, Newtonsoft.Json, Npgsql)
+- **Note**: Tests may fail on systems without .NET 5.0 runtime installed
+
+### Dependencies and Packages
+- **gRPC**: Primary communication protocol (.proto files in AgQueue.Models)
+- **SQLite/PostgreSQL**: Supported database backends
+- **StyleCop.Analyzers**: Code analysis and formatting
+- **MSTest**: Testing framework for integration tests
+- **BenchmarkDotNet**: Performance benchmarking
+- **MessagePack**: Serialization (note: has known vulnerability in current version)
+- **Newtonsoft.Json**: JSON handling (note: has known vulnerability in current version)
+
+## Common Development Tasks
+
+### Adding New Queue Operations
+1. Add method to `InternalApi` class
+2. Implement corresponding gRPC service method in `AgQueueService`
+3. Update storage interface (`IStorage`) if needed
+4. Implement in storage providers (SQLite/Postgres)
+5. Add integration tests
+6. Update documentation
+
+### Adding New Storage Provider
+1. Create new project (e.g., `AgQueue.Redis`)
+2. Implement `IStorage` and `IStorageTransaction` interfaces
+3. Follow patterns from `StorageSqlite` implementation
+4. Handle primary key generation consistently
+5. Add appropriate error handling and resource disposal
+
+### Message Workflows
+1. **Add Message**: CreateQueue → StartTransaction → QueueMessage → CommitTransaction
+2. **Pull Message**: StartTransaction → DequeueMessage → (Process) → CommitTransaction/RollbackTransaction
+3. **Peek Message**: PeekMessageByQueue/PeekMessageById (no transaction required)
+
+### Message Properties
+- **Payload**: Binary message data (bytes)
+- **Priority**: Lower number = higher priority (int32)
+- **MaxAttempts**: Retry limit before message expiration
+- **ExpiryInMinutes**: Message lifetime (0 = no expiration)
+- **CorrelationId**: Optional application-defined correlation (int32)
+- **GroupName**: Optional grouping string
+- **MetaData**: Optional descriptive string data
+
+### Message States
+- **Active**: Available for processing
+- **InTransaction**: Currently being processed
+- **Processed**: Successfully completed
+- **Expired**: Exceeded expiry time
+- **AttemptsExceeded**: Exceeded max retry attempts
+
+### Testing Guidelines
+- Use `[DoNotParallelize]` attribute for integration tests
+- Initialize storage with `DeleteExistingData = true` for clean tests
+- Clean up resources in `TestCleanup` methods
+- Test both success and error scenarios
+
+## Important Notes
+
+### Transaction Complexity
+The codebase has two transaction concepts that can be confusing:
+- **Database transactions**: SQL-level atomicity (IStorageTransaction)
+- **Queue transactions**: Business-level message handling (stored in Transaction table)
+
+### Performance Considerations
+- Int64 primary keys chosen over GUIDs for better database performance
+- Single-process limitation due to application-generated primary keys
+- SQLite chosen for durability over raw speed
+
+### Migration Notes
+- Project originally planned TCP and REST servers
+- Consolidated to gRPC-only for better performance and feature completeness
+- Some legacy documentation may reference old architecture
+
+## StyleCop Configuration
+
+The project uses StyleCop Analyzers with custom rules:
+- SA1200 (using directives placement) is disabled
+- SA1413 (trailing commas) is disabled
+- Company name and copyright are automatically applied
+- License type: MIT
+
+Always ensure your changes conform to the configured StyleCop rules and maintain zero warnings.
\ No newline at end of file
diff --git a/AgQueue.Common/C:/dev/AgQueue/AgQueue.Common/AgQueue.Common.xml b/AgQueue.Common/C:/dev/AgQueue/AgQueue.Common/AgQueue.Common.xml
deleted file mode 100644
index 0c29af7..0000000
--- a/AgQueue.Common/C:/dev/AgQueue/AgQueue.Common/AgQueue.Common.xml
+++ /dev/null
@@ -1,279 +0,0 @@
-
-
-
- AgQueue.Common
-
-
-
-
- Extensions methods for use on Objects.
-
-
-
-
- Throws ArgumentNullException if object is null.
-
- Object to perform null check on.
- Param name to include in exception message.
-
-
-
- Extensions methods to be used on stings.
-
-
-
-
- Standardizes the queue name so it's always consistent.
-
- Raw queue name.
- Standardized queue name.
-
-
-
- Interface for Client to implement.
-
-
-
-
- Current state of a message.
-
-
-
-
- Gets active value. Is the message is live and can be pulled from the queue.
-
-
-
-
- Gets InTransaction. Means the message is currently tied to a transaction, either during insert of during processing. IOW, this message is currently being inserted or pulled from the queue.
-
-
-
-
- Gets processed state. i.e. This message has been processed and will not be pulled.
-
-
-
-
- Gets if Message has expired and will not be pulled.
-
-
-
-
- Gets boolean representing if RetryExceeded. Message retry limit has been reached and message will no longer be pulled.
-
-
-
-
- Represents a Queue Message.
-
-
-
-
- Gets generated unique message id.
-
-
-
-
- Gets the id of the queue.
-
-
-
-
- Gets the id of the queue transaction.
-
-
-
-
- Gets the transaction action. i.e. was this message added or pulled in the transaction.
-
-
-
-
- Gets the datetime the message was added.
-
-
-
-
- Gets the datetime the message was closed, i.e. processed or cancelled or expired.
-
-
-
-
- Gets the priority of the message. Lower is higher priority.
-
-
-
-
- Gets number of attempts to have message processed, i.e. commited.
-
-
-
-
- Gets the number of rollbacks or timeouts before the message expires.
-
-
-
-
- Gets DateTime the message will expire.
-
-
-
-
- Gets the interger correlation id, used by calling application.
-
-
-
-
- Gets string group name. Used by external application for grouping purposes.
-
-
-
-
- Gets actual message data.
-
-
-
-
- Gets Message State, i.e. Active, closed, etc.
-
-
-
-
- Gets string based metadata describing the message in more detail. Optional.
-
-
-
-
- Represents a Queue.
-
-
-
-
- Gets the unique queue id.
-
-
-
-
- Gets the name of the queue.
-
-
-
-
- Represents a Queue Transaction.
-
-
-
-
- Gets the unique ID for a transaction.
-
-
-
-
- Gets a value indicating the state of the transaction.
-
-
-
-
- Gets the date and time the transaction was created.
-
-
-
-
- Gets the date and time the transaction will expire. e.g. after this datetime, the transaction will automatically rollback.
-
-
-
-
- Gets the date and time the transaction was closed, null if not closed.
-
-
-
-
- Represents the action performed within a transaction.
-
-
-
-
- This should never happen.
-
-
-
-
- Item was added within a transaction (if rolledback, delete).
-
-
-
-
- Item was pulled within transaction, increment retry count if rolledback.
-
-
-
-
- Result of transaction commital.
-
- Only commits use the results since commits may have valid responses. If a rollback fails, an exception should be thrown, at least for now.
-
-
-
- Transaction was successfully committed.
-
-
-
-
- Transaction expired before the commit could be completed.
-
-
-
-
- The transaction was already closed.
-
-
-
-
- Transaction could not be found. Perhaps this should be an exception.
-
-
-
-
- The current state of a transaction.
-
-
-
-
- This should never occur. It means there is a serious bug.
-
-
-
-
- Transaction is active.
-
-
-
-
- Transaction has been committed by user.
-
-
-
-
- Transaction was rolled back by user.
-
-
-
-
- Transaction was automatically expired due to timeout.
-
-
-
-
- Exception thrown for unique WorkQueue exceptions.
-
-
-
-
- Initializes a new instance of the class.
-
- Error message.
-
-
-
diff --git a/AgQueue.Postgres/C:/dev/AgQueue/AgQueue.Postgres/AgQueue.Postgres.xml b/AgQueue.Postgres/C:/dev/AgQueue/AgQueue.Postgres/AgQueue.Postgres.xml
deleted file mode 100644
index 311c700..0000000
--- a/AgQueue.Postgres/C:/dev/AgQueue/AgQueue.Postgres/AgQueue.Postgres.xml
+++ /dev/null
@@ -1,137 +0,0 @@
-
-
-
- AgQueue.Postgres
-
-
-
-
- Wrapper around SQLite transaction. Used by storage classes.
-
-
-
-
- Initializes a new instance of the class.
-
- SQLite connection.
-
-
-
- Gets the internal SQLite transaction.
-
-
-
-
-
-
-
-
-
-
- Extensions on the IStorageTransaction interface.
-
-
-
-
- Converts the IStorageTransaction to a NpgsqlTransaction.
-
- The IStorageTransaction to convert.
- SqliteTransaction.
-
-
-
- Implements the IStorage interface for storing and retrieving queue date to SQLite.
-
-
-
-
- Initializes a new instance of the class.
-
- The Sqlite connection string.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Executes an anonymous method wrapped with robust logging, command line options loading, and error handling.
-
- Options class to load from command line.
- The anonymous method execute. Contains a logging object and the options object, both of which can be accessed in the anonymous method.
- Sqlite connection.
- Returns generic object T.
-
-
-
- Executes an anonymous method wrapped with robust logging, command line options loading, and error handling.
-
- Options class to load from command line.
- The anonymous method execute. Contains a logging object and the options object, both of which can be accessed in the anonymous method.
- Sqlite connection.
- Returns generic object T.
-
-
-
diff --git a/AgQueue.Server.Common/C:/dev/AgQueue/AgQueue.Library/AgQueue.Library.xml b/AgQueue.Server.Common/C:/dev/AgQueue/AgQueue.Library/AgQueue.Library.xml
deleted file mode 100644
index 97022fb..0000000
--- a/AgQueue.Server.Common/C:/dev/AgQueue/AgQueue.Library/AgQueue.Library.xml
+++ /dev/null
@@ -1,503 +0,0 @@
-
-
-
- AgQueue.Server.Common
-
-
-
-
- Starting point for accessing all queue related APIS
- This is mostly a factory for creating Queues and Transactions.
-
-
- Exceptions are not used unless there is an exceptional condition. For example, if an items doesn't exist or a param is invalid,
- this is handled without an exception. This is mostly for speed and simplicity with the gRPC interface.
-
-
-
-
- Initializes a new instance of the class.
-
- The storage implementation to use for storage of queues and messages.
- The QueueOptions object passed in via DI.
-
-
-
- Creates a new queue.
-
- The name of the queue.
- A Queue info object. Note: the queueName is the stadnardized name.
-
-
-
- Delete a queue and all messages in the queue.
- Throws an exception if Queue doesn't exist.
-
- Name of the queue to delete.
-
-
-
- Returns information about the requested queue.
-
- The ID of the queue.
- QueueInfo object. Returns null if not found.
-
-
-
- Returns information about the requested queue.
-
- The Name of the queue to lookup.
- QueueInfo object. Null if not found.
-
-
-
- Deletes a queue and 1) rollsback any transaction related to the queue, 2) deletes all messages in the queue.
-
- Queue id.
- ValueTask.
-
-
-
- Disposes of storage resources.
-
-
-
-
- Creates underlying structure in the storage layer.
-
- Will delete and recreate underlying structures.
- Returns ValueTask.
-
-
-
- Starts a transaction used my add message and pull message.
-
- Override default expiration time.
- Transaction ID.
-
-
-
- Extends the transaction by x number of minutes (from the current datetime).
-
- The transaction to extend.
- How long to extend the transaction by.
- ValueTask.
-
-
-
- View the next message in the spcified queue.
-
- Keep in mind, this may change by the time of the dequeue call.
- The queue to peek from.
- Message object or null if no message available.
-
-
-
- Retrieve a message by it's message ID.
-
- Keep in mind, this may change by the time of the dequeue call.
- The ID of the message to retrieve.
- Message object or null if no message available.
-
-
-
- Commits Transaction, updating all messages in transaction.
-
- Transaction Id to commit.
- ValueTask.
-
-
-
- Rollsback a transaction, undoing any changes to messages in the transaction.
-
- The transaction ID of the transaction to rollback.
- ValueTask.
-
-
-
- Adds a message to the specified queue.
-
- The transaction to add the message in.
- The queue the message will be placed in.
- The message payload.
- Message metadata.
- The message priority. Higher numbers will have higher priority.
- How many times should the message pull fail before being retired.
- How long in minutes before the message will expire before it's pulled. 0 = no expiration.
- Correlation number.
- Group name applied to message.
- Return the message id.
-
-
-
- Dequeues a message from a specific queue.
-
- The transaction ID of the transaction the message will be dequeued in.
- The queue to pull the messge from.
- Returns a message object or null if there is no message to dequeue.
-
-
-
- Expires transactions, message and checks message counts, maybe.
-
- The current time is passed in so it is consistent with the time used in the calling procedure.
- ValueTask.
-
-
-
- The interface for storing and retrieving queue information from a storage mechinism, usually a database.
- When implementing IStorage, use the StorageSqlite as an example. There should be no business logic in
- classes that implement IStorage.
-
-
-
-
- Called when Queue process starts. Connections to the storage should be made here, etc.
-
- Should all existing queues and messages be deleted.
- ValueTask.
-
-
-
- Create a new Queue in storage.
-
- Queue name.
- ValueTask.
-
-
-
- Delete a Queue and ALL messages in the Queue.
-
- Queue Id of the queue to delete.
- ValueTask.
-
-
-
- Returns a Transaction object or null value if not found.
-
- The Id of the transaction to lookup.
- Transaction object or null if not found.
-
-
-
- Add a message to the storage.
-
- Queue Transaction ID.
- ID of the storage queue.
- Message data.
- Datetime the message was added.
- String metadata on message data.
- Message priority.
- How many retries before expires.
- Datetime the message will expire.
- Correlation ID.
- Group name.
- Always will be "add".
- "Always will be in transaction.
- Message ID.
-
-
-
- Returns the id and name of the Queue. If no queue is found, returns null.
-
-
- This search should be case sensitive, only use LIKE with SQLite.
-
- Name of the queue to lookup.
- QueueInfo containing ID and Name of queue. Null if not found.
-
-
-
- Returns the id and name of the Queue. If no queue is found, returns null.
-
-
- This search should be case sensitive, only use LIKE with SQLite.
-
- ID of the queue to lookup.
- QueueInfo containing ID and Name of queue. Null if not found.
-
-
-
- Starts a transaction for use when adding or pulling messages.
-
- When the transaction started.
- When the transaction will end.
- Returns transaction ID.
-
-
-
- Updates a set of messages.
-
- The storage transaction to make change within.
- The transaction the messages are associated with.
- The TransactionAction to search for.
- Current message state.
- Update the message state to this value.
- Returns the number of messages updated.
-
-
-
- Updates the message retry cuont based on transactionAction and MessageState.
-
- The storage transaction to run the query under.
- The messages must be in this transaction.
- The transactionAction the message must be in.
- The messageState the message must be in.
- Number of messages updated.
-
-
-
- Delete messages that were added in the specified transtacion.
-
- The storage transaction to run the query in.
- The transaction the messages must be in.
- Number of records deleted.
-
-
-
- Extends the transaction's expiration datetime.
-
- The id of the transaction to update.
- The new expiration datetime.
- ValueTask.
-
-
-
- Update the transaction's state and end datetime.
-
- The storage transaction to make change within.
- The id of the transaction to update.
- The new state of the transaction.
- Reason the transaction was ended. Optional.
- Datetime the transaction was closed (or null if not closed).
- ValueTask.
-
-
-
- Starts a storage (database) transaction, not a queue transaction.
-
-
- Not all Storage classes will have internal transactions, so this can return a dummy class that performs no actions.
-
- Returns a class represented by IStorageTransaction which can commit or rollbacl the transaction.
-
-
-
- Deletes added messages in an expired transaction.
-
- Storage Transaction to run under.
- DateTime to expire against.
- Returns the count of the deleted records.
-
-
-
- Updates the retry counts for messages in an expired transaction.
-
- Storage Transaction to run under.
- DateTime to expire against.
- Number of records updated.
-
-
-
- Expires transactions whose expiry date time is past the currentDateTime.
-
- Storage Transaction to run under.
- DateTime to expire against.
- Returns the number of transactions expired.
-
-
-
- Expire messages that are past their expiration date.
-
- Date used to validate expiration date against. Also to be used as the close datetime.
- Number of messages updated.
-
-
-
- Close message that are beyond their retry count.
-
- DateTime to use a close datetime.
- Number of messages closed.
-
-
-
- Dequeue the next message in the queue and flag the message with the transaction id.
-
- The transaction id to add to the message.
- The queue to pull the message frrom.
- Message object or null of no message to pull.
-
-
-
- View the next message in the queue, without removing it from queue.
-
- The queue to pull the message from.
- Message object or null of no message to pull.
-
-
-
- View a message by the message ID.
-
- The ID of the message to view.
- Message object or null of no message to view.
-
-
-
- Represents a transaction used by the storage (usually a database transaction).
-
-
-
-
- Commits the transaction, usually this reprents a database transaction.
-
-
-
-
- Rollsback the transaction, usually this reprents a database transaction.
-
-
-
-
- Contains information a specific queue.
-
-
-
-
- Gets or sets queue name.
-
-
-
-
- Gets or sets queue ID.
-
-
-
-
- Options used by the queue server.
-
-
-
-
- How many minutes until a message expires. Will be overriden by the value in the message, unless it's zero.
- 0 = max message timeout. 0 in the message, means use the detault here..
-
-
-
-
- How long until a transaction expires in minutes. The value in the transaction will override this value,
- unless it zero. If this is zero as well, transactions won't expire.
-
-
-
-
- Result codes from internal API calls. This currently map directly to gRPC status codes, but
- the internal api may be used with another protocol, hence it can't depend on gRPC status codes.
- Most of these are not used by the Internal API.
-
-
-
-
- Not an error; returned on success.
-
-
-
-
- The operation was cancelled (typically by the caller).
-
-
-
-
- Unknown error. An example of where this error may be returned is if a Status
- value received from another address space belongs to an error-space that is not
- known in this address space. Also errors raised by APIs that do not return enough
- error information may be converted to this error.
-
-
-
-
- Client specified an invalid argument. Note that this differs from FAILED_PRECONDITION.
- INVALID_ARGUMENT indicates arguments that are problematic regardless of the state
- of the system (e.g., a malformed file name).
-
-
-
-
- Deadline expired before operation could complete. For operations that change
- the state of the system, this error may be returned even if the operation has
- completed successfully. For example, a successful response from a server could
- have been delayed long enough for the deadline to expire.
-
-
-
-
- Some requested entity (e.g., file or directory) was not found.
-
-
-
-
- Some entity that we attempted to create (e.g., file or directory) already exists.
-
-
-
-
- The caller does not have permission to execute the specified operation. PERMISSION_DENIED
- must not be used for rejections caused by exhausting some resource (use RESOURCE_EXHAUSTED
- instead for those errors). PERMISSION_DENIED must not be used if the caller can
- not be identified (use UNAUTHENTICATED instead for those errors).
-
-
-
-
- Some resource has been exhausted, perhaps a per-user quota, or perhaps the entire
- file system is out of space.
-
-
-
-
- Operation was rejected because the system is not in a state required for the
- operation's execution. For example, directory to be deleted may be non-empty,
- an rmdir operation is applied to a non-directory, etc.
-
-
-
-
- The operation was aborted, typically due to a concurrency issue like sequencer
- check failures, transaction aborts, etc.
-
-
-
-
- Operation was attempted past the valid range. E.g., seeking or reading past end
- of file.
-
-
-
-
- Operation is not implemented or not supported/enabled in this service.
-
-
-
-
- Internal errors. Means some invariants expected by underlying system has been
- broken. If you see one of these errors, something is very broken.
-
-
-
-
- The service is currently unavailable. This is a most likely a transient condition
- and may be corrected by retrying with a backoff. Note that it is not always safe
- to retry non-idempotent operations.
-
-
-
-
- Unrecoverable data loss or corruption.
-
-
-
-
- The request does not have valid authentication credentials for the operation.
-
-
-
-
diff --git a/AgQueue.Sqlite/C:/dev/AgQueue/AgQueue.Sqlite/AgQueue.Sqlite.xml b/AgQueue.Sqlite/C:/dev/AgQueue/AgQueue.Sqlite/AgQueue.Sqlite.xml
deleted file mode 100644
index c401fab..0000000
--- a/AgQueue.Sqlite/C:/dev/AgQueue/AgQueue.Sqlite/AgQueue.Sqlite.xml
+++ /dev/null
@@ -1,137 +0,0 @@
-
-
-
- AgQueue.Sqlite
-
-
-
-
- Wrapper around SQLite transaction. Used by storage classes.
-
-
-
-
- Initializes a new instance of the class.
-
- SQLite connection.
-
-
-
- Gets the internal SQLite transaction.
-
-
-
-
-
-
-
-
-
-
- Extensions on the IStorageTransaction interface.
-
-
-
-
- Converts the IStorageTransaction to a SqliteTransaction.
-
- The IStorageTransaction to convert.
- SqliteTransaction.
-
-
-
- Implements the IStorage interface for storing and retriving queue date to SQLite.
-
-
-
-
- Initializes a new instance of the class.
-
- The Sqlite connection string.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
- Executes an anonymous method wrapped with robust logging, command line options loading, and error handling.
-
- Options class to load from command line.
- The anonymous method execute. Contains a logging object and the options object, both of which can be accessed in the anonymous method.
- Sqlite connection.
- Returns generic object T.
-
-
-
- Executes an anonymous method wrapped with robust logging, command line options loading, and error handling.
-
- Options class to load from command line.
- The anonymous method execute. Contains a logging object and the options object, both of which can be accessed in the anonymous method.
- Sqlite connection.
- Returns generic object T.
-
-
-