Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions samples/DuckDbFlightSQL/.vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"powerquery.general.mode": "SDK",
"powerquery.sdk.test.settingsFiles": "./Tests/Settings",
"powerquery.sdk.defaultExtension": "${workspaceFolder}\\bin\\AnyCPU\\Debug\\DuckDb.mez",
"powerquery.sdk.defaultQueryFile": "${workspaceFolder}\\DuckDb.query.pq"
}
426 changes: 426 additions & 0 deletions samples/DuckDbFlightSQL/DuckDb.pq

Large diffs are not rendered by default.

28 changes: 28 additions & 0 deletions samples/DuckDbFlightSQL/DuckDb.proj
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="Current" Sdk="Microsoft.Build.NoTargets/3.7.56" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<EnableLocalization>False</EnableLocalization>
</PropertyGroup>
<ItemGroup>
<MEZContent Include="DuckDb.pq" />
<MEZContent Include="FlightSqlAdbcConfig.pqm" />
<MEZContent Include="TypeInfo.pqm" />
<MEZContent Include="SqlGenerator.pqm" />
<MEZContent Include="SqlGeneratorCommon.pqm" />
<MEZContent Include="resources.resx" />
</ItemGroup>
<PropertyGroup>
<MezOutputDir>$(MSBuildProjectDirectory)\bin\AnyCPU\Debug\</MezOutputDir>
<MezStagingDir>$(MSBuildProjectDirectory)\obj\MEZ\</MezStagingDir>
<MezOutputPath>$(MezOutputDir)$(MSBuildProjectName).mez</MezOutputPath>
</PropertyGroup>
<Target Name="PackageMez" AfterTargets="Build" Inputs="@(MEZContent)" Outputs="$(MezOutputPath)">
<RemoveDir Directories="$(MezStagingDir)" />
<MakeDir Directories="$(MezStagingDir)" />
<Copy SourceFiles="@(MEZContent)" DestinationFolder="$(MezStagingDir)" />
<MakeDir Directories="$(MezOutputDir)" Condition="!Exists('$(MezOutputDir)')" />
<ZipDirectory SourceDirectory="$(MezStagingDir)" DestinationFile="$(MezOutputPath)" Overwrite="true" />
<Message Text="Created $(MezOutputPath)" Importance="high" />
</Target>
</Project>
7 changes: 7 additions & 0 deletions samples/DuckDbFlightSQL/DuckDb.query.pq
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
// Use this file to write queries to test your data connector.
// Pressing Evaluate (or Ctrl+E) in VS Code will execute this query
// against the extension defined in .vscode/settings.json.
let
Source = DuckDb.Contents("localhost:31337", "Disabled")
in
Source
31 changes: 31 additions & 0 deletions samples/DuckDbFlightSQL/FlightSqlAdbcConfig.pqm
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
// FlightSqlAdbcConfig.pqm — ADBC driver configuration for DuckDb via FlightSQL.
let
TypeInfo = Extension.LoadExpression("TypeInfo.pqm"),
FlightSqlAdbcConfig = [
Name = "FlightSqlAdbcDuckDb",
Folder = "FlightSQL",
File = "libadbc_driver_flightsql.dll",
DriverType = "Unmanaged",
EntryPoint = "FlightSqlDriverInit",
IdentifierQuoteChar = """",
CatalogNameSeparator = ".",
TableTypes = "BASE TABLE, VIEW",
SupportsCatalog = true,
SupportsSchema = true,
SupportedConnectionOptions = [
adbc.connection.catalog = false,
adbc.connection.db_schema = false
],
TypeInfo = Table.SelectColumns(
TypeInfo,
{"Name", "SqlType", "Unsigned", "ColumnSize", "NumberPrecisionRadix", "LiteralPrefix", "LiteralSuffix"}
)
],
Extension.LoadExpression = (name as text) =>
let
binary = Extension.Contents(name),
asText = Text.FromBinary(binary)
in
Expression.Evaluate(asText, #shared)
in
FlightSqlAdbcConfig
104 changes: 104 additions & 0 deletions samples/DuckDbFlightSQL/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# DuckDb Connector (FlightSQL / ADBC)

A sample Power Query M connector for **DuckDb** using the **FlightSQL** protocol via **ADBC** (Arrow Database Connectivity). Designed as a reference implementation for partners building FlightSQL-based connectors for Power BI.

## Architecture

Uses **`Adbc.Connection`** exclusively — no ODBC, `Adbc.Query`, or `Adbc.Database` needed.

| Feature | Implementation |
| ------------------ | ------------------------------------------------------- |
| **Connection** | `Adbc.Connection` via FlightSQL gRPC |
| **Navigation** | Database → Schema → Table via `information_schema` |
| **Query Folding** | `SqlView.Generator` with DuckDb-specific `SqlGenerator` |
| **Type Mapping** | Full DuckDb coverage (BOOLEAN through STRUCT/MAP) |
| **Relationships** | PK/FK detection via `information_schema` constraints |
| **Authentication** | Username/Password, Bearer Token, Anonymous |
| **DirectQuery** | Supported |
| **Native Query** | Supported with folding |

## FlightSQL ADBC Driver

The FlightSQL ADBC driver (`libadbc_driver_flightsql.dll`) ships with both **Power Query SDK Tools** and **Power BI Desktop**. No separate driver installation is needed.

`FlightSqlAdbcConfig.pqm` configures the driver by specifying:

- **Driver location**: folder (`FlightSQL`), file name, and entry point (`FlightSqlDriverInit`)
- **Metadata**: catalog/schema support, identifier quoting, supported table types (`BASE TABLE`, `VIEW`)
- **Type mapping**: references `TypeInfo.pqm` for DuckDB-to-M type resolution

The connector uses `Adbc.Connection` with `grpc://` (TLS disabled) or `grpc+tls://` (TLS enabled) URIs to establish the FlightSQL gRPC connection.

## Files

```text
DuckDb/
├── DuckDb.pq # Main connector entry point
├── DuckDb.query.pq # Evaluation query for SDK testing
├── FlightSqlAdbcConfig.pqm # ADBC driver configuration
├── TypeInfo.pqm # DuckDb type → M type mapping
├── SqlGenerator.pqm # DuckDb SQL generator overrides
├── SqlGeneratorCommon.pqm # Shared Sql92 base infrastructure
├── DuckDb.proj # Power Query SDK project file
├── resources.resx # Localized strings
└── Tests/
├── Credentials/ # Credential files for test authentication
├── ParameterQueries/ # Parameter query definitions
├── Settings/ # PQTest settings (Sanity, Standard, DatasourceSpecific)
├── TestSuites/ # Test cases organized by category
│ ├── Setup/ # Docker setup, data loading, readme
│ ├── DatasourceSpecific/ # DuckDB-specific tests
│ └── PerfTests/ # Performance test queries
│ # Sanity & Standard tests use shared testframework/tests/TestSuites/
├── RunDuckDbTestsWithVSCodeGuide.md # VS Code test guide
└── RunDuckDbPerfTestsGuide.md # Performance test guide
```

## Key Design Decisions

### Adbc.Connection Only

Simplest approach for FlightSQL — directly creates a connection, builds navigation via SQL metadata queries, and wires up `SqlView.Generator` for folding.

### SQL Generator

The query folding engine uses a three-layer override architecture:

| Layer | File | Role |
| -------------------- | ------------------------ | ------------------------------------------------------------------------------------------------- |
| **SQL92 Base** | `SqlGeneratorCommon.pqm` | Shared infrastructure: type validation, AST helpers, 40+ function stubs, base SQL92 capabilities |
| **DuckDB Overrides** | `SqlGenerator.pqm` | DuckDB dialect: 24 type facets, LIMIT/OFFSET syntax, function remapping, typed literal generation |
| **Connector Wiring** | `DuckDb.pq` | Creates `SqlView.Generator` instance, wires to `Adbc.Connection` via `GetData` |

`SqlGenerator.pqm` loads the base via `Extension.LoadExpression()`, defines an override record, and calls `MergeOverrides("Sql92", Override, false)` to produce the final generator.

**Key DuckDB overrides:**

- **LIMIT/OFFSET**: Translates M's `Table.FirstN`/`Table.Skip` to `LIMIT n OFFSET m`
- **Function remapping**: `TIMESTAMPADD`/`TIMESTAMPDIFF` to DuckDB's `date_add`/`date_diff`, `Text.PositionOf` to `INSTR`, `Text.StartsWith` to `starts_with`
- **Typed literals**: `DATE '2023-01-01'`, `TIMESTAMP '...'`, `TIME '...'`, `CAST(value AS TYPE)` for numeric/string types
- **Type facets**: 24 DuckDB types with `NativeTypeName`, precision, radix, and scale metadata
- **Supported conversions**: Type casting rules (e.g., `BOOLEAN` to `VARCHAR`/`DECIMAL`/`BIGINT`)

Partners building new ADBC connectors can use this as a template: copy `SqlGeneratorCommon.pqm` as-is, then create a dialect-specific `SqlGenerator.pqm` with overrides for their database's SQL syntax and type system.

### Primary & Foreign Keys

Queries `information_schema.table_constraints` + `key_column_usage` for PKs, applies them via `Type.ReplaceTableKeys`. Exposes `GetForeignKeys()` for FK discovery. Power BI uses these to auto-create relationships.

## Supported Types

BOOLEAN, TINYINT, SMALLINT, INTEGER, BIGINT, HUGEINT, FLOAT, DOUBLE, DECIMAL, DATE, TIME, TIMESTAMP, TIMESTAMP WITH TIME ZONE, VARCHAR, CHAR, BLOB, BINARY, UUID, JSON, INTERVAL, ARRAY, STRUCT, MAP

## Testing

Tests require a DuckDB FlightSQL server running locally. See [Test Data Setup](Tests/TestSuites/Setup/readme.md) for Docker container setup and data loading instructions.

- [Running Tests with VS Code](Tests/RunDuckDbTestsWithVSCodeGuide.md) — recommended approach using Test Explorer
- [Running Performance Tests](Tests/RunDuckDbPerfTestsGuide.md) — performance testing with PQPerf

## Building

```powershell
dotnet build DuckDb.proj
```
Loading