Draft
Conversation
* Create cfake.py
* Update __init__.py
* Create cfake.py tests
* update
- simplify & combine code
- adjust 'pattern': use '[^/?#]', match lines and groups
- generalize example URLs
- update default filenames
* update docs/supportedsites
* update test results
---------
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
fixes regression introduced in ab3e34e
* [aryion] Implement search extractor.
* [aryion] Update capabilities.
* [aryion] Adjust example.
* fix flake8 errors
* update & simplify
- use existing '_pagination_next()'
- remove '_pagination_search()'
- update 'search[…]' metadata
* add tests
---------
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
* Add support for sxypix.com
* update & simplify
- text.extract -> text.extr
- self.request().json() -> self.request_json()
- use self.groups
- remove __init__
- list comprehension
* update test results
---------
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
* Add support for ok.porn
* update
- adjust pattern/example/page_url
- prevent unnecessary request
- extract 'tags'
- text.extract -> text.extr
- return list comprehension result
* update supportedsites name
* update test results
---------
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
* Add support for pornstars.tube
* update
- adjust pattern/example/page_url
- extract '' & '' metadata
- prevent unnecessary request
- fix result URLs
- return list comprehension result
* update supportedsites
* update test results
* fix extractor name
---------
Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
- skip 'build_query()' step - add underscores to prefixes
Fixes: ```sh python scripts/init.py newsite https://newsite.com # [INFO] gallery_dl/extractor/newsite.py # [ERROR] TypeError: 'NoneType' object is not iterable # Traceback (most recent call last): # File "/gallery-dl/scripts/init.py", line 58, in init_extractor # fp.write(func(args)) # ^^^^^^^^^^ # File "/gallery-dl/scripts/init.py", line 88, in generate_module # {generate_extractors(args)}\ # ^^^^^^^^^^^^^^^^^^^^^^^^^ # File "/gallery-dl/scripts/init.py", line 108, in generate_extractors_basic # for subcat in args.subcategories: # TypeError: 'NoneType' object is not iterable # [INFO] gallery_dl/extractor/__init__.py # [INFO] test/results/newsite.py ```
* [eporner] Add support
* Line-length <80 chars
* Fix flake8 indentation
* update
- generalize example URL
- strip "Photo Gallery" from 'title'
- extract 'id'
* update supportedsites
* update test results
with the default '"order-posts": "desc"'
with the default '"order-posts": "desc"'
use server invite link that will never expire mikf#8803 (comment)
to allow disabling the behavior introduced in commit a79a945
* adding site support for thefap.com * fixing typo in url tld * improve & simplify 'model' extractor * update 'post' extractor * update docs/supportedsites * add tests --------- Co-authored-by: Mike Fährmann <mike_faehrmann@web.de>
some instances respond with redirects when using these URLs as is
* Add alternate turbovid domain * simplify regex pattern * add tests
bbappserver
force-pushed
the
patch-1
branch
2 times, most recently
from
e774e70 to
1efce7c
Compare
…ckoff Introduced a LearningDelay class to manage API throttling dynamically. Updated the DeviantartOAuthAPI to utilize the new delay mechanism for improved handling of rate limits. Recover frame rate limiting with exponential backoff instead of linear.
Owner
|
By the way, I have plans on completely rewriting the DA module without any OAuth API usage. Soon ™️. So working on any OAuth API features or improvements is kind of pointless. At least in the long run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Deviantart throttling right now i kind of bad and can have quite poor recovery performance.
When the user makes a request within the throttle period the period is increased in a tarpit probably with some exponential factor less than 2 as the DA API doc say you should use 2 as your exp backoff factor.
Current backoff logic does not account for this and moves the delay linearly which causes the user to be repeatedly penalized if they get a 429 for a long time until their throttling value gets huge, because the linear client delay is smaller than the tarpit so they will be penalized until some reset.
There are two proposed changes in this update: