v3.31.4

Reminder

Older Python versions are supported for 3 years after being obsoleted by a new major release. As Python 3.10 was
released on 24 October 2022, the codebase will be streamlined by removing support for Python 3.10 on or after 24 
October 2025.

Added
`````
* Support for Python 3.14.

Fixed
`````
* Fixed ``deepdiff`` differ to handle text strings correctly (e.g. when an API typically returning JSON starts
  returning an error in HTML).

Internals (impacting hooks.py)

• In the Differ Class' process method, the report_kind's value text has been renamed plain for
  clarity and to align with IANA's media type nomenclature for different types of text.

Internals (other)

* Implemented testing for Python 3.14.
* Implemented OpenSSF Scorecard and improved code security.
* Enabled additional ``ruff check`` linters and improved code quality.

v3.31.3

Reminder

Older Python versions are supported for 3 years after being obsoleted by a new major release. As Python 3.10 was
released on 24 October 2022, the codebase will be streamlined by removing support for Python 3.10 on or after 24 
October 2025.

Fixed
`````
* Certain job Exceptions would fail with a yaml Exception.

Internals

• Removed non-unique elements in pyproject.toml's classifiers,
• Updated run-gemini-cli to fix GitHub error.
• Fixed pre-commit.ci failing checks on new PRs

v3.31.2

Fixed

* Fixed UnboundLocalError when using new ``utf-8`` sub-directive within the ``smtp`` emailer (``email`` report).
  Reported in #`104 <https://github.com/mborsetti/webchanges/issues/110>`__.

Internals

• Removed workaround for Python 3.9, which is no longer supported.

v3.31.1.post2

Reminder

Older Python versions are supported for 3 years after being obsoleted by a new major release. As Python 3.10 was
released on 24 October 2022, the codebase will be streamlined by removing support for Python 3.10 on or after 24 
October 2025.

Added
`````
* Documented the ``deepdiff`` differ for side-effects when using ``ignore_order: true``.
* Added the ``utf-8`` configuration sub-directive within the ``smtp`` emailer (``email`` report) to enable turning off 
  RFC 6531 Internationalized Email, aka SMTPUTF8 service extension, for backward compatibility with old SMTP servers.
  Requested in #`108 <https://github.com/mborsetti/webchanges/issues/108>`__.

Fixed
`````
* Fixed regresion: getting messages with deletion when the content still seems to be there. Reported in #`104
  <https://github.com/mborsetti/webchanges/issues/104>`__.
* Fixed handling of Playwright exceptions for ``browser: true`` jobs. Reported in #`106
  <https://github.com/mborsetti/webchanges/issues/106>`__.
* Fixed ``--detailed-versions`` CLI, which raised an Exception if playwright was installed but no Chromium browser was
  available.
* Fixed ``monospace: true`` in HTML reports, which was overly applied to include Comparison Type banners.
* Fixed ``deepdiff`` differ, which would not convert Markdown to HTML when needed.
* Fixed url jobs with ``no_redirects: true``, who would not report the redirect response correctly.
* Fixed ``--delete-snapshot`` CLI, which would not display the snapshot time in the timezone set for reports (if one is
  set).
* Fixed ``xpath`` filter, which would throw an Exception with an XPath ``concat()`` string function.
* Fixed default port for SMTP email to ``port: 587``, as is the correct one given the default ``starttls: true``.

Internals

• Code is now formated with ruff format instead of black.
• Code is now linted with ruff check instead of isort and flake8 (and its extensions).
• Packages in test environments are now installed with uv.
• Experimenting with Gemini CLI GitHub Action <https://github.com/google-github-actions/run-gemini-cli/>__ to triage
  issues and perform pull request reviews (thanks to Google's generous free-of-charge quotas).
• Starting to implement lazy loading of packages and modules to improve startup time to execute simple command line
  arguments.

v3.31.0

⚠ Breaking Changes

* Differ ``ai_google`` (BETA) now expects the API key to be in the environment variable named ``GEMINI_API_KEY`` to
  maintain consistency with the new API documentation from Google and is interoperable with Gemini CLI. The deprecated
  ``GOOGLE_AI_API_KEY`` will be read until the end of 2025.

Added
`````
* Directive ``wait_for_selector`` of URL jobs with ``browser: true`` (Playwright) can now be a list, enabling waiting
  for multiple selectors in the order given.
* Differ ``deepdiff`` has a new sub-directive ``compact`` which produces a much less wordy report in YAML and ignores
  changes of data type (e.g. "type changed from NoneType to str").
* The command-line arguments ``--jobs``, ``--config``, and ``--hooks`` feature a "smart file specification" capability.
  This allows you to provide a shorthand name for your files, and  **webchanges** will automatically search for several
  variations of that name.

Changed
```````
* Differ ``deepdiff``'s report has been improved by indenting multi-line value changes.
* Command line ``--test-job``: Improved display by adding to the report the media type (fka MIME type), ETag (when
  present) and GUID (internal identifier).
* HTML reports now treat the job directive ``note`` as Markdown.

Deprecated
``````````
* Environment variable ``GOOGLE_AI_API_KEY`` for the API key used by differ ``ai_google`` (BETA) is deprecated; use
  ``GEMINI_API_KEY`` instead.

Fixed
`````
* Job directive ``ignore_connection_errors``: Did not work as expected with the default ``httpx`` HTML client library
  (#`100 <https://github.com/mborsetti/webchanges/issues/100>`__.).
* Command line argument ``--error``: Was reporting jobs with sites returning HTTP response code 304 Not Modified as if
  they returned no data (#`102 <https://github.com/mborsetti/webchanges/issues/102>`__).
* Command line argument ``--database``: If filename was not not present in the current directory, it was not searched in
  the default database directory.
* Command line argument ``--test-job``:

  - Output is no longer colorized as if it were a diff.
  - When used in conjunction with ``--test-reporter browser`` or any other HTML reporter and the job has
    ``monospace: true``, the output now uses a monospaced font.
* Configuration ``differ_defaults``: Was not being applied correctly in certain circumstances.
* Differ ``ai_google`` (BETA): Improved reporting of upstream API errors.
* Differ ``images`` (BETA):

  - Report now includes the old image;
  - Minor fixes to the ``ai_google`` summary (ALPHA), including proper application of defaults from the config file
    and the inclusion in the footnote of the actual generative AI model used (vs. the one specified).

Internals / hooks.py

• GUID is now assigned to the job when it's loaded, enabling for a hook to programmatically change job.url,
  job.user_visible_url and/or job.command without causing processing errors downstream.
• Jobs with a directive kind belonging to a Class defined in hooks.py, and which inherits from UrlJob, BrowserJob or
  ShellJob, were not initialized with the default configurations for the parent class (fixed).
• Updated vendored code providing httpx.Headers (when httpx is not installed) to mirror version 0.28.1.
• Updated vendored code providing packaging.versions (when packaging is not installed) to mirror version 24.2.
• Minor code fixes resulting from newly using Pyright / Pylance in the IDE.

v3.30.0

Added

• README links to a new Docker implementation which includes the Chrome browser. Generously offered by and maintained
  by Jeff Hedlund <https://github.com/jhedlund>__ as per #96 <https://github.com/mborsetti/webchanges/issues/96>__.
• New filter jsontoyaml to convert JSON to YAML, which is generally more readable in a report for humans.
• New yaml data type for the deepdiff differ (in addition to json and xml).

Changed

• The deepdiff differ will now try to derive the data-type (when it is not specified) from the data's media
  type (fka MIME type) before defaulting to json.

Fixed

• Fixed confusing warning when no default hooks.py file exists. Thanks to Marcos Alano <https://github.com/mhalano>__
  for reporting in #97 <https://github.com/mborsetti/webchanges/issues/97>__.
• The format-json filter now uses JSON text instead of plain text to report errors caused by it not receiving
  valid JSON data, to be compatible with downstream filters or differs.
• Fixed the differ ai_google (BETA), which under certain circumstances was omitting the footnote with the version
  of the GenAI model used.

v3.29.0

⚠ Breaking Changes

• The differ command now requires that the name: command subdirective of differ be specified.

Changed

• The differ command now has a sub-directive is_html to indicate when output is in HTML format. Thanks to Jeff Hedlund <https://github.com/jhedlund>__ for requesting this enhancement in
  #95 <https://github.com/mborsetti/webchanges/issues/95>__.
• Added a tip in the documentation on how to add bullet points <https://webchanges.readthedocs.io/en/stable/advanced.html#bullet-points>__ to improve the legibility of HTML
  reports.

Fixed

• Fixed reporting of errors arising from filters or reporters.
• Fixed reporting of repeated errors (i.e. when the same error occurs multiple times).
• Fixed header and colorization of the differ command.

v3.28.2

Changed

• The filter format-json will no longer raise an error when it is not fed JSON data to facilitate
  troubleshooting, and will now report the JSONDecodeError details and the full string causing this.

• Documentation for the css and xml filters has been split into two separate entries for ease of reference.

• Minor word editing of error messages to improve clarity.

• Various updates to the differ ai_google (BETA):

  • top_p is set to 1.0 if temperature is 0.0 (its default value) to eliminate randomness in results.
  • Wordsmitting of the default system prompt leading to small improvements.
  • The footnote now shows the model actually used vs. the one specified in the model sub-directive, useful when
    omitting the version when using an experimental version (e.g. specifying gemini-2.0-pro-exp instead of
    gemini-2.0-pro-exp-02-05).

Internals

• Tested the image differ's ai_google directive (ALPHA, undocumented), which uses GenAI to summarize
  differences between two images, with the new gemini-2.0-pro-exp-02-05 experimental <https://ai.google.dev/gemini-api/docs/models/experimental-models#available-models>__ and improved default system
  prompt. While the new model shows improvements by producing a plausible-sounding summary instead of gibberish, the
  summary is highly inaccurate and therefore unusable. Development paused again until model accuracy improves.

v3.28.1

Changed

• Differ ai_google (BETA) now defaults to using the newer gemini-2.0-flash GenAI model, as it performs better.
  Please note that this model "only" handles 1,048,576 input tokens: if you require the full 2M tokens, manually revert
  to using the gemini-1.5-pro GenAI model or try the newer gemini-2.0-pro-exp-02-05 experimental <https://ai.google.dev/gemini-api/docs/models/experimental-models#available-models>__ one.

Fixed

• Fixed bug introduced in 3.28.0 throwing an Exception when reading a configuration file that does not contain the new
  directive differ_defaults. Thanks yubiuser <https://github.com/yubiuser>__ for
  reporting this in issue #93 <https://github.com/mborsetti/webchanges/issues/93>__.

Internals

• When running with -verbose, no longer logs an INFO message for the internal exception raised when receiving a
  an HTTP 304 status code "Not Modified".

refs/tags/v3.28.0

Added

• Added support for setting default differ directives in config.yaml. This is particularly useful for the ai_google
  differ to specify a default GenAI model.
• Added automatic installation of the zstandard <https://github.com/indygreg/python-zstandard>__ library to support
  zstd (RFC 8878 <https://datatracker.ietf.org/doc/html/rfc8878>__) compression in url jobs using the default
  HTTPX HTTP client.

Changed

• Renamed job directives filter and diff_filter to filters and diff_filters (plural nouns) to better
  reflect their list nature. The singular forms remain backward-compatible.
• Consolidated HTTP proxy configuration into a single proxy directive, replacing the separate http_proxy and
  https_proxy directives while maintaining backward compatibility.
• Improved maximum parallel executions of use_browser: true to ensuring each Chrome instance has at least 400 MB
  of available memory (or the maximum available, if lower).

Fixed

• Fixed handling of "Error Ended" reports to only send them with suppress_repeated_errors: true.
• Fixed error message when using job directive http_client: requests without the requests <https://pypi.org/project/requests/>__ library installed. Thanks yubiuser <https://github.com/yubiuser>__ for
  reporting this in issue #90 <https://github.com/mborsetti/webchanges/issues/90>__.
• Improved and standardized lthe ogic and documentation for the use of environment variables HTTPS_PROXY and
  HTTP_PROXY in proxy settings.
• Modified --prepare-jobs command line argument to append never run jobs to command line jobs (joblist), if
  present, rather than replacing them.

Internals

• Replaced JobBase attributes http_proxy and https_proxy with a unified proxy attribute.
• Updated JobBase attributes from singular filter and diff_filter to plural filters and diff_filters.
• Removed unused JobBase attribute chromium_revision (deprecated since Pypetteer removal on 2022-05-02).

Changes in version 3.27.0

(inadvertently omitted from release notes)

Added

• Python 3.13: webchanges is now fully tested on Python 3.13 before releasing. However, orderedset, a dependency
  of the aioxmpp <https://pypi.org/project/aioxmpp/>__ library required by the xmpp reporter will not install in
  Python 3.13 (at least on Windows) and this reporter is therefore not included in the tests. It appears that the
  development of this library <https://codeberg.org/jssfr/aioxmpp>__ has been halted.

  • Python 3.13t (free-threaded, GIL-free) remains unsupported due to the lack of free-threaded wheels for dependencies
    such as cryptography, msgpack, lxml, and the optional jq.

• New job directive suppress_repeated_errors to notify an error condition only the first time it is encountered. No
  more notifications will be sent unless the error resolves or a different error occurs. This enhancement was
  requested by toxin-x <https://github.com/toxin-x>__ in issue #86 <https://github.com/mborsetti/webchanges/issues/86>__.

• New command line argument --log-file to write the log to a file. Suggested by yubiuser <https://github.com/yubiuser>__ in issue #88 <https://github.com/mborsetti/webchanges/issues/88>__.

• pypdf filter has a new extraction_mode optional sub-directive to enable experimental layout text extraction
  mode functionality.

• New command-line option --prepare-jobs to run only newly added jobs (to capture and save their initial snapshot).

Fixed

• Fixed command line argument --errors to use the same exact logic as the one used when running webchanges.
  Reported by yubiuser <https://github.com/yubiuser>__ in issue #88 <https://github.com/mborsetti/webchanges/issues/88>__.
• Fixed incorrect reporting of job error when caused by an HTTP response status code that is not IANA-registered <https://docs.python.org/3/library/http.html#http-status-codes>__.

Changed

• Command line --test can now be combined with --test-reporter to have the output sent to a different reporter.
• Improved error reporting, including reporting error message in --test and adding proxy information if the error
  is a network error and the job has a proxy and.
• Updated the default model instructions for the ai_google (BETA) differ to improve quality of summary.

Internals

• Now storing error information in snapshot database.
• Added ai_google directive to the image differ to test Generative AI summarization of changes between two
  images, but in testing the results are unusable. This feature is in ALPHA and undocumented, and will not be
  developed further until the models improve to the point where the summary becomes useful.