Changes¶

0.6.0 (2024-05-29)¶

Improved how the default retry policy handles temporary download errors. Before, 3 HTTP 429 responses followed by a single HTTP 520 response would have prevented a retry. Now, unrelated responses and errors do not count towards the HTTP 520 retry limit.
Improved how the default retry policy handles network errors. Before, after 15 minutes of unsuccessful responses (e.g. HTTP 429), any network error would prevent a retry. Now, network errors must happen 15 minutes in a row, without different errors in between, to stop retries.
Implemented an optional aggressive retry policy, which retries more errors more often, and could be useful for long crawls or websites with a low success rate.
Improved the exception that is raised when passing an invalid retrying policy object to a Python client.

RequestError now has a query attribute with the Zyte API request parameters that caused the error.

ZyteAPI and AsyncZyteAPI sessions no longer need to be used as context managers, and can instead be closed with a close() method.

Removed Python 3.7 support.
Added ZyteAPI and AsyncZyteAPI to provide both sync and async Python interfaces with a cleaner API.
Deprecated zyte_api.aio:
- Replace zyte_api.aio.client.AsyncClient with the new AsyncZyteAPI class.
- Replace zyte_api.aio.client.create_session with the new AsyncZyteAPI.session method.
- Import zyte_api.aio.errors.RequestError, zyte_api.aio.retry.RetryFactory and zyte_api.aio.retry.zyte_api_retrying directly from zyte_api now.
When using the command-line interface, you can now use --store-errors to have error responses be stored alongside successful responses.
Improved the documentation.

Include the Zyte API request ID value in a new .request_id attribute in zyte_api.aio.errors.RequestError.

w3lib >= 2.1.1 is required in install_requires, to ensure that URLs are escaped properly.
unnecessary requests library is removed from install_requires
fixed tox 4 support

Fixed an issue with submitting URLs which contain unescaped symbols
New “retrying” argument for AsyncClient.__init__, which allows to set custom retrying policy for the client
--dont-retry-errors argument in the CLI tool

Connections are no longer reused between requests. This reduces the amount of ServerDisconnectedError exceptions.

Bump minimum aiohttp version to 3.8.0, as earlier versions don’t support brotli decompression of responses
Declared Python 3.11 support

Network errors, like server timeouts or disconnections, are now retried for up to 15 minutes, instead of 5 minutes.

Require to install Brotli as a dependency. This changes the requests to have Accept-Encoding: br and automatically decompress brotli responses.

Internal AggStats class is cleaned up:

AggStats.n_extracted_queries attribute is removed, as it was a duplicate of AggStats.n_results
AggStats.n_results is renamed to AggStats.n_success
AggStats.n_input_queries is removed as redundant and misleading; AggStats got a new AggStats.n_processed property instead.

This change is backwards incompatible if you used stats directly.

aiohttp.client_exceptions.ClientConnectorError is now treated as a network error and retried accordingly.
Removed the unused zyte_api.sync module.

Temporary download errors are now retried 3 times by default. They were not retried in previous releases.

This release contains usability improvements to the command-line script:

Instead of python -m zyte_api you can now run it as zyte-api;
the type of the input file (--intype argument) is guessed now, based on file extension and content; .jl, .jsonl and .txt files are supported.