Skip to content

Commit

Permalink
Allow linking to settings Scrapy-style (#212)
Browse files Browse the repository at this point in the history
  • Loading branch information
Gallaecio authored Dec 13, 2024
1 parent d7a54ed commit 6145931
Show file tree
Hide file tree
Showing 3 changed files with 105 additions and 56 deletions.
25 changes: 25 additions & 0 deletions docs/_ext/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
def setup(app):
# https://stackoverflow.com/a/13663325
#
# Scrapy’s
# https://github.com/scrapy/scrapy/blob/dba37674e6eaa6c2030c8eb35ebf8127cd488062/docs/_ext/scrapydocs.py#L90C16-L110C6
app.add_crossref_type(
directivename="setting",
rolename="setting",
indextemplate="pair: %s; setting",
)
app.add_crossref_type(
directivename="signal",
rolename="signal",
indextemplate="pair: %s; signal",
)
app.add_crossref_type(
directivename="command",
rolename="command",
indextemplate="pair: %s; command",
)
app.add_crossref_type(
directivename="reqmeta",
rolename="reqmeta",
indextemplate="pair: %s; reqmeta",
)
3 changes: 3 additions & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import pkgutil
import sys
from datetime import datetime
from pathlib import Path

sys.path.insert(0, os.path.abspath("../"))

Expand Down Expand Up @@ -56,7 +57,9 @@ def get_version_and_release():
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
sys.path.insert(0, str(Path(__file__).parent.absolute())) # _ext
extensions = [
"_ext",
"sphinx.ext.autodoc",
"sphinx.ext.intersphinx",
"sphinx.ext.ifconfig",
Expand Down
133 changes: 77 additions & 56 deletions docs/settings.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,38 +7,39 @@ Configuring the settings denoted below would follow the usual methods used by
Scrapy.


SCRAPY_POET_PROVIDERS
---------------------
.. setting:: SCRAPY_POET_CACHE

Default: ``{}``
SCRAPY_POET_CACHE
-----------------

A ``dict`` wherein the **keys** would be the providers available for your Scrapy
project while the **values** denotes the priority of the provider.
Default: ``None``

More info on this at this section: :ref:`providers`.
The caching mechanism in the **providers** can be enabled by either setting this
to ``True`` which configures the file path of the cache into a ``.scrapy/`` dir
in your `local Scrapy project`.

On the other hand, you can also set this as a ``str`` pointing to any path relative
to your `local Scrapy project`.

SCRAPY_POET_OVERRIDES
---------------------

Deprecated. Use ``SCRAPY_POET_RULES`` instead.
.. setting:: SCRAPY_POET_CACHE_ERRORS

SCRAPY_POET_RULES
-----------------
SCRAPY_POET_CACHE_ERRORS
------------------------

Default: :meth:`web_poet.default_registry.get_rules()
<web_poet.rules.RulesRegistry.get_rules>`
Default: ``False``

Accepts a ``List[ApplyRule]`` which sets the rules to use.
When this is set to ``True``, any error that arises when retrieving dependencies from
providers would be cached. This could be useful in cases during local development
wherein you outright know that retrieving the dependency would fail and would
choose to ignore it. Caching such errors would reduce the waiting time when
developing Page Objects.

.. warning::
It's `recommended` to set this off into ``False`` by default since you might miss
out on sporadic errors.

Although ``SCRAPY_POET_RULES`` already has values set from the return value of
:meth:`web_poet.default_registry.get_rules() <web_poet.rules.RulesRegistry.get_rules>`,
make sure to also set the ``SCRAPY_POET_DISCOVER`` setting below.

There are sections dedicated for this at :ref:`intro-tutorial` and
:ref:`rules-from-web-poet`.
.. setting:: SCRAPY_POET_DISCOVER

SCRAPY_POET_DISCOVER
--------------------
Expand All @@ -60,54 +61,29 @@ in ``SCRAPY_POET_DISCOVER`` (e.g. when the annotated page objects are inside you
Scrapy project). However, it's recommended to still use ``SCRAPY_POET_DISCOVER``
to ensure all the intended rules are properly loaded.

SCRAPY_POET_CACHE
-----------------

Default: ``None``

The caching mechanism in the **providers** can be enabled by either setting this
to ``True`` which configures the file path of the cache into a ``.scrapy/`` dir
in your `local Scrapy project`.

On the other hand, you can also set this as a ``str`` pointing to any path relative
to your `local Scrapy project`.


SCRAPY_POET_CACHE_ERRORS
------------------------

Default: ``False``

When this is set to ``True``, any error that arises when retrieving dependencies from
providers would be cached. This could be useful in cases during local development
wherein you outright know that retrieving the dependency would fail and would
choose to ignore it. Caching such errors would reduce the waiting time when
developing Page Objects.

It's `recommended` to set this off into ``False`` by default since you might miss
out on sporadic errors.

.. setting:: SCRAPY_POET_OVERRIDES

SCRAPY_POET_TESTS_DIR
SCRAPY_POET_OVERRIDES
---------------------

Default: ``fixtures``
Deprecated. Use ``SCRAPY_POET_RULES`` instead.

Sets the location where the ``savefixture`` command creates tests.

More info at :ref:`testing`.
.. setting:: SCRAPY_POET_PROVIDERS

SCRAPY_POET_PROVIDERS
---------------------

SCRAPY_POET_TESTS_ADAPTER
-------------------------
Default: ``{}``

Default: ``None``
A ``dict`` wherein the **keys** would be the providers available for your Scrapy
project while the **values** denotes the priority of the provider.

Sets the class, or its import path, that will be used as an adapter in the
generated test fixtures.
More info on this at this section: :ref:`providers`.

More info at :ref:`fixture-adapter`.

.. setting:: SCRAPY_POET_REQUEST_FINGERPRINTER_BASE_CLASS

SCRAPY_POET_REQUEST_FINGERPRINTER_BASE_CLASS
--------------------------------------------
Expand All @@ -127,3 +103,48 @@ for requested dependencies. Otherwise, the fingerprint is used as is.
serialized with :func:`repr` for fingerprinting purposes. If you find a
real-world scenario where this is a problem, please `open an issue
<https://github.com/scrapinghub/scrapy-poet/issues>`_.


.. setting:: SCRAPY_POET_RULES

SCRAPY_POET_RULES
-----------------

Default: :meth:`web_poet.default_registry.get_rules()
<web_poet.rules.RulesRegistry.get_rules>`

Accepts a ``List[ApplyRule]`` which sets the rules to use.

.. warning::

Although ``SCRAPY_POET_RULES`` already has values set from the return value of
:meth:`web_poet.default_registry.get_rules() <web_poet.rules.RulesRegistry.get_rules>`,
make sure to also set the ``SCRAPY_POET_DISCOVER`` setting below.

There are sections dedicated for this at :ref:`intro-tutorial` and
:ref:`rules-from-web-poet`.


.. setting:: SCRAPY_POET_TESTS_ADAPTER

SCRAPY_POET_TESTS_ADAPTER
-------------------------

Default: ``None``

Sets the class, or its import path, that will be used as an adapter in the
generated test fixtures.

More info at :ref:`fixture-adapter`.


.. setting:: SCRAPY_POET_TESTS_DIR

SCRAPY_POET_TESTS_DIR
---------------------

Default: ``fixtures``

Sets the location where the ``savefixture`` command creates tests.

More info at :ref:`testing`.

0 comments on commit 6145931

Please sign in to comment.