v0.12.0

Added

• Fedwire (Legacy Wire) File Support (e7e2189): Complete legacy Fedwire message file support (Experimental)
  • File format: Tag-value text format (.fed) used by the Federal Reserve's large-value real-time gross settlement system
  • Flat table structure: All FEDWireMessage fields (~326 columns) flattened into a single {filename}_message table with 1 row per file
  • All fields as TEXT: Wire format stores amounts as fixed-width strings; all columns use TEXT type to preserve formatting
  • Full round-trip support: Parse → query/modify via SQL → export back to valid .fed format
  • Registry-backed TableSet: registerWireTableSet / getWireTableSet / UnregisterWireTableSet / ClearWireTableSetRegistry for managing original Wire structures needed for round-trip export
  • WireTableInfo struct: Provides MessageTable() and AllTableNames() methods for programmatic table name discovery
  • GetWireTableInfos(): Returns []WireTableInfo for all registered Fedwire files
  • IsWireBaseTableName(): Checks if a table name matches the Fedwire _message suffix convention
  • DumpFedWire() / DumpFedWireWithTableSet(): Export Fedwire tables from database back to .fed files
  • OutputFormatFedWire: New output format enum for auto-save and dump operations
  • ErrWire: Sentinel error for Fedwire operation failures
  • FileTypeFedWire: New file type constant with String(), extension(), baseType() support
  • Streaming support: streamFedWireFileToDatabase() for file-path input, streamWireFileToDatabase() for io.Reader input
  • AddFS support: Fedwire files in fs.FS (including embed.FS) are properly detected and loaded
  • Auto-save integration: performFedWireAutoSave() and overwriteOriginalFiles() handle .fed files alongside ACH and tabular formats
  • Test coverage: Unit tests for file detection, parsing, registry, SQL queries, round-trip, and export

Fixed

• Windows file lock in AddFS (e7e2189): Added closer: file to readerInput in file_processor.go so that FS-opened files are properly closed after streaming. Previously, TempDir RemoveAll cleanup failed on Windows because files remained open
• Dead code removal in builder.go (e7e2189): Removed unused processFSToReaders method (~85 lines) and deduplicateCompressedFiles wrapper from DBBuilder. The actual code path uses fileProcessor.processFSToReaders(). Updated builder_test.go accordingly

Changed

• Documentation Updates: Added Fedwire Support sections to all README files (7 languages: EN, ES, FR, JA, KO, RU, ZH-CN)
  • Supported formats table updated with .fed extension
  • Experimental status warning, table structure explanation, TEXT column rationale, limitations, security considerations, and code examples
• Auto-save cleanup refactoring: Renamed cleanupACHRegistry() to cleanupTableSetRegistries() to handle both ACH and Fedwire registry cleanup on connection close
• ACH table detection improvement: dumpSQLiteDatabase() now verifies registry presence before treating _message suffix tables as ACH, preventing false positives with Fedwire's _message tables

Dependencies

• github.com/nao1215/fileparser: v0.4.0 → v0.5.1 (adds Wire subpackage for Fedwire parsing)
• github.com/moov-io/wire: v0.15.7 (new indirect dependency via fileparser)

v0.11.0

Added

• JSON / JSONL File Support (e45329d): Complete JSON and JSON Lines file format support with SQLite json_extract() integration
  • JSON format: Array root → one row per element, Object root → single row. Raw JSON stored in data TEXT column
  • JSONL format: One row per line with bufio.Reader (no line size limit). Empty lines silently skipped, invalid lines rejected with line number
  • Query with json_extract(): Access nested fields via SQLite's built-in JSON functions
    • Example: SELECT json_extract(data, '$.name') FROM my_table
    • Example: SELECT json_extract(data, '$.address.city') FROM my_table
  • Compression support: All 8 compression formats supported for both JSON and JSONL (.json.gz, .json.bz2, .json.xz, .json.zst, .json.z, .json.snappy, .json.s2, .json.lz4, and corresponding .jsonl.* variants)
  • Streaming chunk processing: processJSONInChunks uses json.Decoder to stream array elements one at a time without loading the entire array into memory, preventing OOM for large JSON files. processJSONLInChunks provides true line-by-line streaming
  • Trailing data validation: Rejects malformed JSON with trailing garbage after array (e.g., [{"a":1}] garbage)
  • 18 new FileType constants: FileTypeJSON, FileTypeJSONL, plus 16 compressed variants
  • Test coverage: Unit tests, integration tests with json_extract() queries, compressed format tests (gzip, zstd, snappy, s2, lz4, zlib), 85.0% overall coverage

Fixed

• Missing parserFileType mappings (e45329d): Added missing zlib, snappy, s2, lz4 mappings for CSV, TSV, LTSV, Parquet, XLSX in parser_bridge.go. Previously these compressed variants would fall through to Unsupported
• Pre-existing lint issues: Fixed prealloc warnings in builder.go and file.go, removed unused colType parameter from newColumnInfoWithType

Changed

• Documentation Updates: Added JSON/JSONL sections to all README files (7 languages: EN, ES, FR, JA, KO, RU, ZH-CN)
  • Supported formats table updated with .json and .jsonl base formats and all compression variants
  • Usage examples with json_extract() queries for flat and nested JSON structures
• Test coverage for parserFileType: Added 38 test cases covering ZLIB, SNAPPY, S2, LZ4 for all existing formats plus all 18 JSON/JSONL mappings

Dependencies

• github.com/nao1215/fileparser: v0.3.1 → v0.4.0 (adds JSON/JSONL parsing support)
• modernc.org/sqlite: 1.40.1 → 1.45.0
• github.com/klauspost/compress: 1.18.2 → 1.18.4
• github.com/pierrec/lz4/v4: 4.1.22 → 4.1.25

v0.10.0

Added

• Custom Logger Support: Flexible logging system with slog integration
  • Logger interface: Simple logging interface with Debug, Info, Warn, Error, and With methods
  • ContextLogger interface: Extended logging interface with context-aware methods (DebugContext, InfoContext, WarnContext, ErrorContext)
  • NewSlogAdapter(): Adapter to use standard library slog.Logger with filesql's Logger interface
  • NewSlogContextAdapter(): Adapter for context-aware logging with slog.Logger
  • WithLogger(): Builder method to inject custom logger into the build and open process
  • nopLogger: Zero-overhead no-op logger implementation used as default (benchmarked at ~0.2 ns/op)
  • Logging throughout build, validation, and database opening operations
  • Comprehensive test coverage and benchmarks for all logger implementations

Changed

• Documentation Updates: Added Custom Logger section to all README files (7 languages: EN, ES, FR, JA, KO, RU, ZH-CN)
  • Usage examples with slog integration
  • Logger and ContextLogger interface definitions
  • Performance benchmark comparison table

v0.9.0

Added

• Read-Only Database Mode: New ReadOnlyDB wrapper for safe read-only access to databases

  • NewReadOnlyDB(db): Wraps existing *sql.DB to prevent write operations
  • ReadOnlyDB.Query(), QueryContext(), QueryRow(), QueryRowContext(): Read operations work normally
  • ReadOnlyDB.Exec(), ExecContext(): Returns ErrReadOnly for write operations (INSERT, UPDATE, DELETE, DROP, ALTER, CREATE, TRUNCATE, REPLACE, UPSERT)
  • ReadOnlyDB.Prepare(), PrepareContext(): Rejects preparation of write statements
  • ReadOnlyDB.Begin(), BeginTx(): Returns ReadOnlyTx for read-only transactions
  • ReadOnlyDB.Ping(), PingContext(), Close(), DB(): Standard database operations
  • ReadOnlyStmt: Read-only prepared statement wrapper
  • ReadOnlyTx: Read-only transaction wrapper with same protections
  • DBBuilder.OpenReadOnly(ctx): Convenience method to open database in read-only mode
  • ErrReadOnly: Sentinel error for rejected write operations
  • Useful for audit scenarios where data viewing without modification risk is required

• ACHTableInfo Struct: New struct for managing ACH table name information

  • ACHTableInfo.BaseName: The base table name derived from ACH filename
  • ACHTableInfo.FileHeaderTable(): Returns {baseName}_file_header
  • ACHTableInfo.BatchesTable(): Returns {baseName}_batches
  • ACHTableInfo.EntriesTable(): Returns {baseName}_entries
  • ACHTableInfo.AddendaTable(): Returns {baseName}_addenda
  • ACHTableInfo.IATBatchesTable(): Returns {baseName}_iat_batches
  • ACHTableInfo.IATEntriesTable(): Returns {baseName}_iat_entries
  • ACHTableInfo.IATAddendaTable(): Returns {baseName}_iat_addenda
  • ACHTableInfo.AllTableNames(): Returns all possible table names for the base name
  • GetACHTableInfos(): Returns []ACHTableInfo for all registered ACH files

Changed

• Internal ACH Function: Made GetACHBaseTableNames private (getACHBaseTableNames) as it was only used internally
  • Use GetACHTableInfos() for public access to ACH table information

v0.8.0

Added

• New Compression Formats: Added support for 4 new compression formats via fileparser v0.2.0
  • zlib (.z) - Standard DEFLATE compression
  • snappy (.snappy) - Google's high-speed compression
  • s2 (.s2) - Improved Snappy extension, faster
  • lz4 (.lz4) - Extremely fast compression

v0.7.0

Changed

• Migrated from internal github.com/nao1215/filesql/parser to external github.com/nao1215/fileparser for file parsing
• Updated all internal references from parser. to fileparser.

Removed

• Internal parser package (now using github.com/nao1215/fileparser v0.1.0 as external dependency)

v0.6.0

Added

• Public Parser Package (6271e5ef): Exposed the internal parser as a public API for use in external projects
  • New parser package: Standalone file parsing without SQLite dependency
    • parser.Parse(): Parse CSV, TSV, LTSV, XLSX, and Parquet files from io.Reader
    • parser.DetectFileType(): Automatic file type detection from file path
    • parser.BaseFileType(): Get base file type from potentially compressed file types
  • Type exports: TableData, ColumnType, FileType types for working with parsed data
  • Parquet support: Full Parquet parsing with parser/parquet.go
  • XLSX support: Excel file parsing with parser/xlsx.go
  • Comprehensive test coverage: 90%+ coverage for the parser package
• ORM Integration Examples (281ede2): Added example code for popular Go ORMs and query builders
  • GORM: Full GORM integration example with model definitions
  • Bun: Bun ORM example with struct scanning
  • Ent: Facebook's Ent framework example with generated code
  • sqlx: sqlx example with struct tags
  • sqlc: sqlc example with generated type-safe queries
  • Squirrel: Squirrel query builder example
  • Basic: Standard library database/sql example
  • Multi-format: Example combining CSV, TSV, and LTSV files
• FileType.String() Method: Added fmt.Stringer implementation for FileType enum
  • Human-readable format names for logging and debugging
  • Returns names like "CSV", "TSV", "LTSV", "XLSX", "Parquet", etc.

Changed

• Documentation Updates: Enhanced README files across all 7 languages
  • Added fileprep project reference (e3705a7)
  • Fixed project link formatting (dea615e)
  • Updated fileprep section (d7905b0)

Technical Details

• Architecture: Parser package enables lightweight file parsing without database overhead
• Compatibility: Parser package can be used independently of the main filesql package
• Testing: Added comprehensive test suites for parser, types, and error handling

v0.5.0

Added

• Benchmark Tests (2852ea2): Added benchmark infrastructure for performance testing
  • New make benchmark target in Makefile for running benchmark tests
  • Benchmark tests isolated with //go:build benchmark tag to prevent execution during regular tests
  • BenchmarkOpenContext and BenchmarkOpenContextParallel for measuring CSV loading performance

Improved

• Major Performance Optimization (d20b3c8, e95a5bf): Significantly improved file loading performance
  • 55% faster execution: Reduced 100,000-row CSV loading time from ~960ms to ~430ms
  • 12% less memory: Reduced memory usage from ~161MB to ~141MB
  • Transaction batching: Wrapped all INSERT operations in a single transaction to reduce SQLite disk sync operations
  • Slice reuse: Pre-allocate and reuse value slices in insertChunkData() to reduce allocations
  • Pre-allocation in type inference: Optimized newColumnInfoList() and inferColumnsInfo() with pre-allocated column value slices

Fixed

• Data Integrity in Chunk Insertion (b191d93): Fixed potential data corruption issues in insertChunkData()
  • Stale value prevention: Fixed issue where records with fewer columns than headers could retain stale values from previous rows
  • Extra column detection: Added validation to fail fast when records have more columns than headers, preventing silent data truncation

Changed

• Documentation Updates (17a42fa): Added benchmark results to all README files (7 languages)
  • Performance metrics: ~430ms execution time, ~141MB memory for 100,000-row CSV

Dependencies

• github.com/klauspost/compress: 1.18.1 → 1.18.2

v0.4.6

Added

• Header-Only File Support (PR #67, 5de8801): Files with headers but no data records are now supported
  • CSV, TSV, Parquet, and XLSX formats can now be loaded with only header rows
  • Creates empty SQLite tables with correct column names (all columns as TEXT type)
  • Useful for schema definition files and template files
  • Example: A CSV file containing only id,name,age will create a table with those columns but zero rows

Fixed

• LTSV Error Handling: Improved error messages for invalid LTSV data
  • Now correctly returns "no valid LTSV keys found" error instead of silently creating empty tables
  • LTSV format requires key:value pairs, so header-only concept does not apply

Changed

• Dependencies: Updated library dependencies
  • modernc.org/sqlite: 1.40.0 → 1.40.1
  • github.com/klauspost/compress: 1.18.0 → 1.18.1
  • github.com/xuri/excelize/v2: 2.9.1 → 2.10.0
  • golang.org/x/crypto: Security update
  • actions/checkout: 4 → 6

v0.4.5

Fixed

• Table Name Sanitization: Fixed SQL syntax errors caused by special characters in file names
  • Applied sanitizeTableName() to all table name generation paths
  • Hyphens, spaces, and special characters are now automatically converted to underscores
  • Example: "user-data.csv" → table "user_data", "my file.csv" → table "my_file"
  • Updated test expectations to match sanitized table names

Improved

• API Documentation: Enhanced documentation for public APIs to clarify table name sanitization
  • Updated Open(), OpenContext(), and DBBuilder.Open() method documentation
  • Added examples showing special character conversion in table names
  • Improved sanitizeTableName() function documentation with detailed transformation rules
• Development Experience: Optimized test execution time for local development
  • Added GitHub Actions environment checks to skip slow tests locally
  • Reduced local test execution time by 63% (from ~55s to ~20s)
  • Maintained full test coverage in CI/CD while improving developer productivity

Technical Details

• Breaking Change Prevention: Preserved existing tableFromFilePath() behavior for backward compatibility
• Test Coverage: Maintained 80.7% test coverage with updated test expectations
• Performance: No impact on runtime performance, only development-time improvements