Conversation
|
Those tests do take forever to complete.. Their site seems to be responding super slow; I'll check/retry again later. |
|
I removed one test that takes over 1h to complete, but still getting issues with the glad dataset. All tests pass locally so I'm not exactly sure what is happening... We will need to investigate a bit more. |
|
It looks like GRIIDC blocked the GitHub's IP. We are getting a 403 error. |
|
Good to go! |
There was a problem hiding this comment.
Pull Request Overview
This PR fixes an issue with the GLAD dataset download by updating the URL and improving the HTTP request configuration. The changes enhance the reliability of downloads by mimicking browser behavior and increasing timeout values.
- Updates the GLAD dataset URL to remove the deprecated
/pelagos-symfonypath segment - Adds browser-like User-Agent headers to HTTP requests to avoid bot detection
- Increases timeout values for better reliability with large file downloads
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| clouddrift/adapters/glad.py | Updates the GLAD dataset URL to the current API endpoint |
| clouddrift/adapters/utils.py | Adds browser headers and increases timeout values for HTTP requests |
| tests/adapters/utils_test.py | Updates test assertions to match the new headers and timeout parameters |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| _BROWSER_HEADERS = { | ||
| "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" |
There was a problem hiding this comment.
[nitpick] The hardcoded User-Agent string may become outdated over time. Consider using a more generic or library-generated User-Agent, or add a comment explaining why this specific version is required.
| _BROWSER_HEADERS = { | |
| "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" | |
| "User-Agent": requests.utils.default_user_agent() |
selipot
left a comment
There was a problem hiding this comment.
I see the modifications, and they seem to do the trick! Thank you for spending the time to fix this!
Uh oh!
There was an error while loading. Please reload this page.