1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
|
=================
Advanced concepts
=================
.. highlight:: python
Namespace packages
==================
Hierarchical package structure
------------------------------
Traditionally, Python packages were organized into a hierarchical
structure with modules and subpackages being located inside the parent
package directory. When submodules are imported, they are represented
as attributes on the parent module. Consider the following session::
>>> import sphinx.addnodes
>>> sphinx
<module 'sphinx' from '/usr/lib/python3.8/site-packages/sphinx/__init__.py'>
>>> sphinx.addnodes
<module 'sphinx.addnodes' from '/usr/lib/python3.8/site-packages/sphinx/addnodes.py'>
This works fine most of the time. However, it start being problematic
when multiple Gentoo packages install parts of the same top-level
package. This may happen e.g. with some plugin layouts where plugins
are installed inside the package. More commonly, it happens when
upstream wishes all their packages to start with a common component.
This is the case with Zope framework. Different Zope packages share
common ``zope`` top-level package. ``dev-python/zope-interface``
installs into ``zope.interface``, ``dev-python/zope-event``
into ``zope.event``. For this to work using the hierarchical layout,
a common package has to install ``zope/__init__.py``, then other Zope
packages have to depend on it and install sub-packages inside that
directory. As far as installed packages are concerned, this is entirely
doable.
The real problem happens when we wish to test a freshly built package
that depends on an installed package. In that case, Python imports
``zope`` from build directory that contains only ``zope.interface``.
It will not be able to import ``zope.event`` that is installed in system
package directory::
>>> import zope.interface
>>> zope
<module 'zope' from '/tmp/portage/dev-python/zope-interface-4.7.1/work/zope.interface-4.7.1-python3_8/lib/zope/__init__.py'>
>>> zope.interface
<module 'zope.interface' from '/tmp/portage/dev-python/zope-interface-4.7.1/work/zope.interface-4.7.1-python3_8/lib/zope/interface/__init__.py'>
>>> import zope.event
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'zope.event'
Now, this could be worked around by copying all other subpackages back
to the build directory. However, there is a better solution.
Namespace package structure
---------------------------
Unlike traditional packages, namespace packages act as a kind of proxy.
They are not strictly bound to the containing directory, and instead
permit loading subpackages from all directories found in module search
path. If we make ``zope`` a namespace package, we can import both
the locally built ``zope.interface`` and system ``zope.event``
packages::
>>> import zope.interface
>>> import zope.event
>>> zope
<module 'zope' (namespace)>
>>> zope.interface
<module 'zope.interface' from '/tmp/portage/dev-python/zope-interface-4.7.1/work/zope.interface-4.7.1-python3_8/lib/zope/interface/__init__.py'>
>>> zope.event
<module 'zope.event' from '/usr/lib/python3.8/site-packages/zope/event/__init__.py'>
There are three common methods of creating namespace packages:
1. `PEP 420`_ namespaces implemented in Python 3.3 and newer,
2. Using pkgutil_ standard library module,
3. Using `namespace package support in setuptools`_ (discouraged).
PEP 420 namespaces are created implicitly when a package directory
does not contain ``__init__.py`` file. While earlier versions
of Python (including Python 2.7) ignored such directories and did not
permit importing Python modules within them, Python 3.3 imports such
directories as namespace packages.
pkgutil namespaces use ``__init__.py`` with the following content::
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
setuptools namespace can use ``__init__.py`` with the following
content::
__import__('pkg_resources').declare_namespace(__name__)
Alternatively, setuptools normally installs a ``.pth`` file that is
automatically loaded by Python and implicitly injects the namespace
into Python.
Both pkgutil and setuptools namespaces are portable to all versions
of Python.
More general information on the topic can be found under `packaging
namespace packages`_ in Python Packaging User Guide.
Determining whether namespaces are used
---------------------------------------
The exact method of detecting namespace packages depends on the type
of namespace used.
PEP 420 namespaces can generally be recognized by the lack
of ``__init__.py`` in an installed package directory. However, since
they do not require any specific action, distinguishing them is not very
important.
pkgutil namespaces can be recognized through the content of their
``__init__.py``. Generally, you should find it suspicious if is
the only file in a top-level package directory, and if the name of this
directory is less specific than the package name (e.g. ``zope`` for
``zope.interface``, ``ruamel`` for ``ruamel.yaml``). If you miss this,
then you will learn about the namespace from package collisions
on the respective ``__init__.py``.
setuptools namespaces usually do not install ``__init__.py`` but
do install a ``.pth`` file instead. The distutils-r1 eclass detects
this automatically and prints a warning. Prior to installation,
they can also be recognized by ``namespace_packages`` option
in ``setup.py`` or ``setup.cfg``.
Adding new namespace packages to Gentoo
---------------------------------------
If the package uses PEP 420 namespaces, no special action is required.
Per PEP 420 layout, the package must not install ``__init__.py`` files
for namespaces.
If the package uses one of the other layouts, their respective files
must be removed from the install tree.
For pkgutil namespace, its ``__init__.py`` should be removed after
the PEP 517 build phase:
.. code-block:: bash
python_compile() {
distutils-r1_python_compile
rm "${BUILD_DIR}/install$(python_get_sitedir)"/jaraco/__init__.py || die
}
The equivalent code for the legacy eclass mode is:
.. code-block:: bash
python_install() {
rm "${BUILD_DIR}"/lib/jaraco/__init__.py || die
distutils-r1_python_install
}
For setuptools namespace, the ``.pth`` file should be removed instead:
.. code-block:: bash
python_compile() {
distutils-r1_python_compile
find "${BUILD_DIR}" -name '*.pth' -delete || die
}
The setuptools code for the legacy mode is:
.. code-block:: bash
python_install_all() {
distutils-r1_python_install_all
find "${D}" -name '*.pth' -delete || die
}
Some packages include an explicit ``setuptools`` runtime dependency
(``install_requires``) when using namespaces. If this is the only
use of ``pkg_resources`` and ``setuptools`` in installed package
sources, this dependency needs to be stripped, e.g.:
.. code-block:: bash
src_prepare() {
# strip rdep specific to namespaces
sed -i -e "/'setuptools'/d" setup.py || die
distutils-r1_src_prepare
}
Legacy namespace packages in Gentoo
-----------------------------------
Historically, Gentoo has used ``dev-python/namespace-*`` packages
to support namespaces. This method is deprecated and it is in process
of being retired.
.. _PEP 420: https://www.python.org/dev/peps/pep-0420/
.. _pkgutil: https://docs.python.org/3/library/pkgutil.html
.. _namespace package support in setuptools:
https://setuptools.readthedocs.io/en/latest/setuptools.html#namespace-packages
.. _packaging namespace packages:
https://packaging.python.org/guides/packaging-namespace-packages/
|
GitWeb