Constant 500 error when querying

[kimjensen90]

Hi there,

I have had a tremendous amount of issues getting my queries to work using this script. Authentication works, and the partitions are create fine, but when the queries are sent to the I'm getting the following:
[NETWORK] Retry attempt for partition 1/2 (43.8s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (42.1s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).

I'm keeping my queries to a minimum, although I would like to get all M365Usage too, I'm testing with Copilot Interactions and only a single day.

I'm fearing that the error could come from many different things - would you have any obvious things to check for?

And now the job failed again - seems to be related to timing: don't know if the query runs for too long:

Job 18 error: [CREATE-FAILED] Network outage exceeded 30 minute tolerance during query creation
  [RETRY-QUEUE] Partition 2/2 marked as Failed for retry (from monitoring loop)
[09.08.54] Total Queries: 2 | Completed: 0 | Remaining: 1 | Failed: 1
  Initial job execution complete. States: Failed:2
    Partition 1 error: [NETWORK] Retry attempt for partition 1/2 (43.8s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (103.3s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (147.6s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (202s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (257.2s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (315.6s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (371.2s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (421.6s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (481.1s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (524.4s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (566.8s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (623.2s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (680.7s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (734s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (780.5s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (825.1s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (869.5s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (926.8s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (973.3s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1027.6s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1082.8s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1129.2s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1184.4s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1240.8s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1297.1s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1343.4s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1393s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1450.4s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1508.2s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1549.3s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1593.5s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1637.9s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1690.3s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1742.7s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 1/2 (1795.1s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[CREATE-FAILED] Partition 1/2 - Network outage exceeded 30 minute tolerance (30.9m elapsed) - will retry at end of run
    Partition 2 error: [NETWORK] Retry attempt for partition 2/2 (42.1s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (102.2s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (158.6s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (203.8s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (254s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (294.5s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (352.8s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (396s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (440.3s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (493.6s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (537.7s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (595.6s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (636s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (679.9s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (735.3s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (785s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (841.4s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (899.9s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (956.1s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1005.5s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1058.6s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1109.8s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1150s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1195.4s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1246.6s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1309.9s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1373.7s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1428.3s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1484.2s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1525.2s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1583.6s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1630.1s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1672.5s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1715s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[NETWORK] Retry attempt for partition 2/2 (1756.4s elapsed) : Response status code does not indicate success: 500 (Internal Server Error).
[CREATE-FAILED] Partition 2/2 - Network outage exceeded 30 minute tolerance (30.2m elapsed) - will retry at end of run
  [RETRY] Pass 2/5 - 2 partition(s) need retry

Final Error:

Below are the API Permissions:

Hope to get some help.

[Rance9]

Hi Kim,

Thanks for the screenshots and detailed information. Quick read on what happened: PAX never actually got a single query created — both partitions are in the "Never Sent" bucket, and every attempt to create the audit query failed for ~43 minutes straight before the script gave up. That's PAX behaving correctly (it's not going to loop forever), but it doesn't tell us why the create call kept failing.

One thing worth flagging: the [NETWORK] label and "Network outage" wording in PAX is just our umbrella bucket for any sustained transport or 5xx failure on the create-query call. It doesn't necessarily mean your network is down — could be Microsoft's audit-query service having a bad stretch, could be a proxy/firewall on your side, could be a token issue. We just need a couple data points to narrow it down.

A few things to try, in order — stop at the first one that clears it:

1. Check Service Health. M365 admin center → Health → Service health, look for anything under Microsoft Purview, Microsoft 365 suite, or Entra around the failure window. If something's listed, that's your answer — wait it out and re-run.

2. Try a different time window. If the manual probe works but PAX keeps failing on a specific date range, shift it by 24 hours. Occasionally a specific window overlaps a stuck partition on the backend and a different window steps right around it.

3. Test the endpoint manually, outside the script. This is the most important one — it tells us instantly whether the problem is on Microsoft's side, your egress, or something script-related. Save the following script code to a .ps1 file, and run it using PowerShell 7+ (use the pwsh command trigger, and not powershell) from the same machine PAX failed on. You will need to update the TenantId, ClientId, and ClientSecret to match the app registration you have currently set up.

if (-not (Get-Module -ListAvailable Microsoft.Graph.Authentication)) {
    Install-Module Microsoft.Graph.Authentication -Scope CurrentUser -Force
}
Import-Module Microsoft.Graph.Authentication -ErrorAction Stop

$TenantId     = '<your-tenant-id>'
$ClientId     = '<your-app-registration-client-id>'
$ClientSecret = '<your-client-secret>'

$secureSecret = ConvertTo-SecureString $ClientSecret -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential($ClientId, $secureSecret)

Connect-MgGraph -TenantId $TenantId -ClientSecretCredential $cred -NoWelcome -ErrorAction Stop

$ctx = Get-MgContext
if (-not $ctx) { throw "Connect-MgGraph failed - no context" }
"Connected as : $($ctx.ClientId)"
"AuthType     : $($ctx.AuthType)"
"AppName      : $($ctx.AppName)"

$nowUtc = (Get-Date).ToUniversalTime()
$body = @{
    displayName         = "pax-manual-probe-$($nowUtc.ToString('yyyyMMddHHmmss'))"
    filterStartDateTime = $nowUtc.AddDays(-1).ToString('yyyy-MM-ddTHH:mm:ssZ')
    filterEndDateTime   = $nowUtc.ToString('yyyy-MM-ddTHH:mm:ssZ')
    operationFilters    = @('CopilotInteraction')
} | ConvertTo-Json -Depth 5

"Request body:"; $body

try {
    $resp = Invoke-MgGraphRequest -Method POST `
        -Uri 'https://graph.microsoft.com/beta/security/auditLog/queries' `
        -Body $body -ContentType 'application/json' `
        -ResponseHeadersVariable h -StatusCodeVariable s -ErrorAction Stop

    "HTTP Status       : $s"
    "request-id        : $($h.'request-id')"
    "client-request-id : $($h.'client-request-id')"
    "queryId           : $($resp.id)"
    "status            : $($resp.status)"
}
catch {
    "EXCEPTION TYPE    : $($_.Exception.GetType().FullName)"
    "Message           : $($_.Exception.Message)"

    if ($_.ErrorDetails -and $_.ErrorDetails.Message) {
        "Response body     :"
        $_.ErrorDetails.Message
    }

    if ($_.Exception.Response) {
        "HTTP status       : $([int]$_.Exception.Response.StatusCode)"
        try {
            "request-id        : $(($_.Exception.Response.Headers.GetValues('request-id'))        -join ',')"
            "client-request-id : $(($_.Exception.Response.Headers.GetValues('client-request-id')) -join ',')"
        } catch { }
    }
    else {
        "(No HTTP response - failure happened before reaching the server: DNS / TLS / proxy / connection.)"
    }
}

• Returns a queryId → service is healthy now, just re-run PAX.
• Returns 5xx → confirmed Microsoft service-side. Capture the request-id for a support ticket.
• Returns 401/403 → auth/permissions issue, send me the code.
• Throws a transport/TLS/DNS exception → something in your egress path (proxy, firewall, TLS inspection).

4. If step 3 returns 5xx, open a Microsoft support case with: tenant ID, UTC timestamp of a failed request, the request-id from the manual probe, and the fact that PAX saw 43 minutes of POST /beta/security/auditLog/queries failures with zero successful creates. They can correlate the request ID against their service logs.

If you would like further help troubleshooting with these suggested steps, or if step 3 returns something we haven't listed in the troubleshooting steps, please reach out again and we'll be glad to continue assisting. We'll leave this GitHub issue open for a few days to allow you some time to respond as needed.

Thanks!
Microsoft Copilot Growth ROI Advisory Team

[kimjensen90]

Hi @Rance9,

Thanks for the swift reply. I have tried 1 and 2 a few times. No messages and the error is just persistent.

Just tested the probe script - I have tested something similar previously, which was able to get passed the error, but still ran for a very long time.

The result I got now is this:

So I'm guessing that I'm looking at a support ticket to be created? I cannot help but notice the specific error, which says something about a data format? Is that a path to follow or just something random?

[Rance9]

Hi Kim,

Thanks for the follow up. It may be something in the test script we sent possible, we're looking into that and will get back with you shortly. It may be that a support ticket will be required, as we have many customers using this script without issues, but before we have you go through that process we want to be sure that's the correct route. Thanks for your patience.

[kimjensen90]

Hi @Rance9 ,

I have also used the PAX script without any issues other than it runs for quite some time when I added the M365Usage to it. I'm part of a rather large tenant, so I do expect that to run for a while. But the Copilot Interactions usually comes rather quickly - especially when I'm getting them for a single day.

Last week I also made my own probe script, which seemed to establish the connections successfully. I requested an even smaller amount of data, which started to come through but I only got 150 records within an hour. Could my Purview/Audit log be overburdened somehow?

Thanks again for your support!

[Rance9]

Hi Kim,

Thanks for running the probe and sharing the result — that screenshot is actually really helpful, and it tells us pretty clearly what's going on. Good news: it's not something you're doing wrong, and it's not your network or your app registration. The not-as-good news: it's something only Microsoft can fix from our side. Let me walk you through what we're seeing.

What the error is actually telling us

The probe script connected successfully, sent a clean, well-formed request to Microsoft, and Microsoft's service came back with HTTP 500 — UnknownError. An HTTP 500 means "the server tried to handle this and crashed internally" — it's the server raising its hand and admitting it had the problem, not you.

The piece you noticed about the data format (Could not convert string to DateTime) is a real clue, but it's a clue pointing at Microsoft's backend, not yours. Here's the important part: the request your probe sent contained perfectly valid, properly-formatted timestamps. We can rule out the request itself — if your timestamps were the problem, you'd get a different error (HTTP 400, "BadRequest"), and you'd get it instantly. Instead you got a 500, and the error message references Newtonsoft.Json, which is a Microsoft-internal software library running on our servers — not anything in PAX or in the probe script or anywhere on your machine.

What's actually happening: when you ask Microsoft's audit-query service to create a new query, it has to read some data from its own internal records before it can respond. One of those records (on Microsoft's side, for your tenant) has a date value stored in it that the service can't read back. So it crashes before it ever gets to your request. Every attempt fails the same way for the same reason. That's why retrying doesn't help, that's why PAX kept failing for 43 minutes straight, and that's why your own probe script hit the exact same wall.

The fact that you also saw painfully slow returns last week (150 records in an hour on a small request) fits the same picture — your tenant's audit-query system seems to be in a degraded state on Microsoft's side. It's not that the audit log itself is full or "overburdened" the way a database can be — it's that the query-management layer that PAX (and your probe, and even the Purview portal under the hood) all talk to is stuck on something.

What this means for next steps

This needs a Microsoft support case. Our support teams have access to the server-side logs and can reset the stuck state on their side. You and I can't reach those internals from outside.

When you open the case, give them:

• Your tenant ID
• The full error JSON from your probe screenshot — especially these four values, which Microsoft uses to look the request up in their internal logs:
  • request-id: 683b33ae-1f01-45cc-8486-5cad8966caa9
  • client-request-id: f38bda64-1a8a-489d-8040-347cb6ed726d
  • innerError.date: 2026-05-04T19:47:27
  • error message: Newtonsoft.Json.JsonReaderException — Could not convert string to DateTime
• The endpoint: POST https://graph.microsoft.com/beta/security/auditLog/queries
• A short description: "Every attempt to create an audit query for this tenant returns HTTP 500 with Newtonsoft.Json.JsonReaderException — Could not convert string to DateTime. The request body is well-formed (verified with a minimal manual probe). Read-side throughput on the same endpoint is also severely degraded (~150 records/hour on a small query last week). Suspect the tenant's audit-query state is in a degraded condition and needs to be cleared on the service side."
• Severity B is appropriate (a production capability is broken but you have other ways to operate.)

A reasonable severity to request is Sev B. When you talk to the engineer, it's worth specifically asking them to:

1. Look up the failed request using the request-id above and pull the server-side exception detail.
2. Check whether your tenant has stuck or orphaned audit-query records that are breaking the create path, and clear them if so.

That's the kind of language that gets the ticket routed to the right team faster than just "I'm getting an error."

One small thing you can try while you wait (optional)

If you're curious whether the broken state is specific to Copilot-related queries or affects everything for your tenant, you can re-run the probe script with one tiny change: in the operationFilters line, swap 'CopilotInteraction' for 'FileAccessed'. Everything else stays the same. If that one also fails the same way, it confirms the problem is tenant-wide (and that's a useful detail to mention in the support ticket). If it succeeds, the problem is scoped to the Copilot side, which is also useful to mention. Either way it doesn't fix anything — it's just an extra data point if you feel like sending it along with the case.

And one offer

When Microsoft Support replies, the engineer's response can sometimes be pretty technical or vague — things like "we cleared the affected partitions" or "the issue was caused by a malformed entry in the query manifest." If you'd like, forward me whatever they send back and I'm happy to translate it into plain English and confirm whether you should be all set to re-run PAX, or whether there's any follow-up worth asking them about. We'll leave this GitHub issue open for a couple of weeks so you have time to get the case opened and hear back without feeling rushed.

To recap the headline so it doesn't get lost in the detail above:

• Your setup is fine. PAX, your app registration, your network — none of those are the problem.
• Microsoft's audit-query service is in a bad state for your tenant and is crashing on its own internal data when you (or PAX) ask it to create a query.
• A support case is the path forward. They can fix it; we can't.
• Forward me the support response when it comes in and I'll help you make sense of it.

Sorry it turned out to be the un-fun answer rather than something we could solve here directly — but at least now we know exactly what to tell Microsoft so they can get it sorted quickly.

[kimjensen90]

HI @Rance9 ,

Thank you so much for the provided information and guidance! We're creating a support ticket as we speak.

Will get back once I get response from them.

Thank you!

/Kim

[Rance9]

Hi @kimjensen90 ,

We just wanted to check in to see if you were able to get your issue resolved through a support case.

Have a great week!

[kimjensen90]

Hi @Rance9 ,

Thanks for following up! Unfortunately, we haven't seen any progress on our tickets that's been open for 2 weeks now. Any chance you could push some buttons internally? Case number is: Case 2605051420001999

Any help would be much appreciated!

/Kim

[Rance9]

Hi Kim,

Thanks for the update, and we're sorry to hear the case has been stalled for that long.

We want to be transparent that we don’t have access to Microsoft’s internal support systems (ICM/DfM), so we're not able to directly view or escalate your case internally through our analytics team. That said, here are a few actions that typically help move these forward:

• If you have a Microsoft account team (CSA/Account Executive), we strongly recommend escalating through them — they can raise internal priority or chase the support engineer directly
• In the M365 Admin Center, you can reply to the case or request an escalation/update — even just asking for a status/RCA can sometimes trigger movement
• If the case is business-impacting, make sure the severity is appropriately set (or request it be increased if needed)

From our side, we're absolutely happy to continue helping on anything PAX-related in parallel — if there’s information or logs you’re waiting on from support, let us know and we can help validate or troubleshoot independently where possible.

Appreciate your patience here, and feel free to keep us posted.