This relaxes the limits for dbcounter to make it flush stats to the
database less often. Currently every thirty seconds or 100 hits, we
write a stats line to the database. In some services (like keystone)
this can trigger more than one write per second because of the
massive number of SELECT calls that service makes.
This removes the hit limit and decreases the mandatory flush interval
to once a minute. Hopefully this will manifest as lower load on the
database triggered by what would be readonly operations.
Change-Id: I43a58532c0541075a2d36408abc50a41f7994bda
There are two problems with dbcounter installation on Jammy. The first
is straightforward. We have to use `py_modules` instead of `modules` to
specify the source file. I don't know how this works on other distros
but the docs [0] seem to clearly indicate py_modules does this.
The second issue is quite an issue and requires story time. When
pip/setuptools insteall editable installs (as is done for many of the
openstack projects) it creates an easy-install.pth file that tells the
python interpreter to add the source dirs of those repos to the python
path. Normally these paths are appended to your sys.path. Pip's isolated
build env relies on the assumption that these paths are appeneded to the
path when it santizes sys.path to create the isolated environemnt.
However, when SETUPTOOLS_SYS_PATH_TECHNIQUE is set to rewrite the paths
are not appended and are inserted in the middle. This breaks pip's
isolated build env which broke dbcounter installations. We fix this by
not setting SETUPTOOLS_SYS_PATH_TECHNIQUE to rewrite. Upstream indicates
the reason we set this half a decade ago has since been fixed properly.
The reason Jammy and nothing else breaks is that python3.10 is the first
python version to use pip's isolated build envs by default.
I've locally fiddled with a patch to pip [1] to try and fix this
behavior even when rewrite is set. I don't plan to push this upstream
but it helps to illustrate where the problem lies. If someone else would
like to upstream this feel free.
Finally this change makes the jammy platform job voting again and adds
it to the gate to ensure we don't regress again.
[0] https://docs.python.org/3/distutils/sourcedist.html#specifying-the-files-to-distribute
[1] https://paste.opendev.org/show/bqVAuhgMtVtfYupZK5J6/
Change-Id: I237f5663b0f8b060f6df130de04e17e2b1695f8a
The mysql performance_schema method for counting per-database queries
is very heavyweight in that it requires full logging (in a table) of
every query. We do hundreds of thousands in the course of a tempest
run, which ends up creating its own performance problem.
This changes the approach we take, which is to bundle a very tiny
sqlalchemy plugin module which counts just what we care about in
a special database.
It is more complex than just enabling the features in mysql, but it
is a massively smaller runtime overhead. It also provides us the
opportunity to easily zero the counters just before a tempest run.
Change-Id: I361bc30bb970cdaf18b966951f217862d302f0b9
