OpenDev is preparing to remove centos-7 nodes on March 15[*]. This
change drops one nodeset definition which is the last remaining
reference on DevStack's master branch. Also clean up a lingering
reference to swift-dsvm-functional-py3 which no longer exists.

[*] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/message/A2YIY5L7MVYSQMTVZU3L3OM7GLVVZPLK/

Change-Id: Icd487e1012263a9b0bc13b529d31ff2025108adf
(cherry picked from commit de9a15ad)

Instead of using RDO Trunk repo server, CentOS official mirrors provide
a most reliable infrastructure and supports EMS which is required when
enabling FIPS in C9S.

RDO didn't publish official release rpms for Xena and Wallaby, install
"rdo-release-yoga" release rpm for Xena. Devstack only uses RDO repository for binary dependencies such as rabbitmq, openvswitch.

Change 0da88c4a is squashed in this
commit.

Closes-Bug: #2020661
Closes-Bug: #2020434
Change-Id: Icd99f467d47aaafaaf3ee8f2a3c4da08842cb672
(cherry picked from commit b2ad00cb)
(cherry picked from commit 4062cc0f)

Devstack set the env var TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE
which are used to use the constraints during Tempest virtual env installation.
Those env var are set to non-master constraint when we need to use non-master
constraints but when we need to use the master constraints we do not set/reset
them point to master constraints. This create the issue when running the grenade
job where we run Tempest on the old devstack as well as in the new devstack.
When tempest is installed on old devstack then old tempest is used and it sets
these env var to stable/<branch> constraints (this is the case when old devstack
(the stable branch is in EM phase) uses the old tempest not the master tempest),
all good till now. But the problem comes when in the same grenade script run
upgrade-tempest install the master tempest (when new devstack branches are in
the 'supported' phase and use the master tempest means) and are supposed to use
the master constraints. But the TOX_CONSTRAINTS_FILE/UPPER_CONSTRAINTS_FILE env
var set by old tempest is used by the tempest and due to a mismatch in constraints
it fails.

This happened when we tried to pin the stable/wallaby with Tempest 29.0.0
- https://review.opendev.org/c/openstack/devstack/+/871782

and table/xena grenade job failed (stable/xena use master tempest and supposed
to use master constraints)
- https://zuul.opendev.org/t/openstack/build/fb7b2a8b562c42bab4c741819f5e9732/log/controller/logs/grenade.sh_log.txt#16641

We should set/reset those constraint env var to master constraints if configuration
tell devstack to use the master constraints.

Making openstacksdk-functional-devstack job non voting as this is
failing 100% on <=stable/xena, depends-On fix that

Depends-On: https://review.opendev.org/c/openstack/python-openstackclient/+/872341

[1] https://github.com/openstack/devstack/blob/71c3c40c269a50303247855319d1d3a5d30f6773/lib/tempest#L124

Closes-Bug: #2003993
Change-Id: I5e938139b47f443a4c358415d0d4dcf6549cd085
(cherry picked from commit 7fe99810)
(cherry picked from commit 30a7d790)
(cherry picked from commit febdb122)
(cherry picked from commit cb891834)

Stable/wallaby is in Extended maintenance state[1] and
Tempest master stopped supporting the stable/wallaby[2] so
we need to pin tempest in stable/wallaby testing.

We use Tempest 29.0.0 compatible tag for wallaby testing.
Due to constraints of teststool 2.4.0 in stable/wallaby[3],
Tempest>=30.0.0 (which removed the workaround of testtool
issue[4]) fail on stable/wallaby with SkipTest are treated
with error[5]. That is why we need to use Temepst 29.0.0
which is compatible with testtool 2.4.0.

Making openstacksdk-functional-devstack job non voting as this is
failing 100% on <=stable/xena
- https://zuul.openstack.org/builds?job_name=openstacksdk-functional-devstack&branch=stable%2Fxena&branch=stable%2Fwallaby&skip=0

Depends-On: https://review.opendev.org/c/openstack/tempest/+/871781
Depends-On: https://review.opendev.org/c/openstack/devstack/+/871945

[1] https://releases.openstack.org/
[2] https://review.opendev.org/c/openstack/tempest/+/864371
[3] https://github.com/openstack/requirements/blob/35c167b04cd0d210503de1ebf7be165d0236b905/upper-const>
[4] https://github.com/testing-cabal/testtools/issues/272
[5] https://zuul.opendev.org/t/openstack/build/0a42465205484a72adaab7d8a116d823/log/job-output.txt#2>

Change-Id: I9a8fad5c1f13107dcd6c23c281bae6a42a10294f

We pinned tox<4 in ensure-tox pre.yaml playbook
- https://review.opendev.org/q/I9a138af94dedc0d8ce5a0d519d75779415d3c30b

but did not realize that it is used in run-both.yaml playbook
also where we should pin too.

Change-Id: I4407f7036e3fd51ac79f68791151c3f9cd03bd01
(cherry picked from commit 689b399e3a445c47368b9a17e428ab120f591d7b)
(cherry picked from commit ff4cd115cc437e966db3717e1b94ff536e71b39e)
(cherry picked from commit f895418f860309d9fc7c7acc8afa578739667520)

Tox 4.0.0 has some incompatible changes, epecially more
strict on allowlist_externals. Tempest recently changed
allowlist_externals not to be *[1] causing the failure
on jobs where lib/tempest failing to run the tempest
as command in virtual env.

----------
venv: commands[0]> tempest verify-config -uro /tmp/tmp.qH5KgJHTF4
venv: failed with tempest is not allowed, use allowlist_externals to allow it
------

We do not need to test/fix the <=stable/zed branches with tox 4.0.0
and pinning them with the compatible tox version of the time stable
brnaches were releaased is better way.

This commit proposes:
1. Pinning the tox<4.0.0 for <=stable/ze branches testing
2. Workaround to unblock the master gate by pinning it <4.0.0 but
we should make our testing compatible with tox 4.0.0 soon.

Depends-On: https://review.opendev.org/c/openstack/devstack/+/867070

Related-Bug: #1999183

[1] https://review.opendev.org/c/openstack/tempest/+/865314 devstack based job started failing to run tempest command on venv.

Change-Id: I9a138af94dedc0d8ce5a0d519d75779415d3c30b
(cherry picked from commit ba54baa4253f2c770760b59a4a8f28317e5e260d)
(cherry picked from commit a3227ba0c0a6b951b301e556f78e634e6ce9e770)
(cherry picked from commit 395f44708506b30b29b35dd655dddd434bac9a08)
(cherry picked from commit 02f286a80a478d662e079fe066d311e4cf7264d8)

Glance started moving to new RBAC and glance-tempest-plugin
and Tempest need to set few configuration to enable the
scope checks on glance side and so does on Temepst side to tell
glance is ready with scope checks so that test can be run with
scoped token.

Conflicts:
     lib/tempest

Change-Id: I09f513d08212bc80a3a86a750b29b1c6625d2f89
(cherry picked from commit 8c930492)

Already n-v devstack-platform-centos-9-stream has been failing
last 2 months.

Considering that there isn't a lot of traffic on stable/wallaby
and we have a centos job in newer branches so any backport
patch gets tested there, the commit removes this job no to
waste CI resources.

Change-Id: Id218cf2ad5fbb9cae5828972366fe85911e500b2

The official Ubuntu cloud images have some further python pkgs
preinstalled that conflict with our requirements. Allow to
overwrite them.

Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Closes-Bug: 1871485
Change-Id: I793c250cae5e7b9bc835b7016d790d1f9ae8a7f3
(cherry picked from commit 61a37bff)

In case of online mode, there is a procedure to recreate tempest venv.
For consistency of tempest venv during the entire stack.sh process,
add logic to consider the TEMPEST_VENV_UPPER_CONSTRAINTS option here.

Closes-bug: #1980483
Signed-off-by: June Yi <june.yi@samsung.com>
Change-Id: I0cea282152fd363af8671cab1b5f733ebe2bd4df
(cherry picked from commit 8355e8130652da0a6ed87295998200d0bcb9a58e)

This patch includes changes required to run devstack on CentOS Stream 9
which has been already published in official repos by CentOS team [1]:

- Add RDO deps repository for CS9.
- remove xinetd package from installation for swift. Note that
  rsync-daemon is installed which should work fine.
- Replace genisoimage by xorriso in CS9.
- Use /etc/os-release to identify the distro in CS9 as it doesn't
  provide lsb_release command.
- Use pip from rpm package instead of from get-pip.py as done in Fedora.
- Add non-voting job devstack-platform-centos-9-stream to the check
  pipeline.

Depends-On: https://review.opendev.org/c/openstack/devstack/+/848365
Change-Id: Ic67cddabd5069211dc0611994b8b8360bcd61bef
(cherry picked from commit 5ea4c3c1)

Newer versions of openssl (CentOS9Stream for example) do not like using sha1.
Devstack will fail on these systems[1] with the following error:
801B93DCE77F0000:error:03000098:digital envelope routines:do_sigver_init:invalid digest:crypto/evp/m_sigver.c:333:
This patch updates the tls-proxy code in devstack to use sha256 instead of sha1 which allows devstack to complete when tls-proxy is enabled.

[1] https://zuul.opendev.org/t/openstack/build/1d90b22a39c74e24a8390861b3c5f957/log/job-output.txt#5535

Closes-Bug: #1962600

Change-Id: I71e1371affe32f070167037b0109a489d196bd31
(cherry picked from commit 35bc600d)

This patch reduces memory usage on the Cinder Volume and Backup services
by tuning glibc.

The specific tuning consist on disabling the per thread arenas and
disabling dynamic thresholds.

The Cinder Backup service suffers from high water mark memory usage and
uses excessive memory.  As an example just after 10 restore operations
the service uses almost 1GB of RAM and does not ever free it afterwards.
With this patch the memory consumption of the service is reduced down to
almost 130MB.  If we add a revert from Cinder (Change-Id
I43a20c8687f12bc52b014611cc6977c4c3ca212c) it goes down to 100MB during
my tests.

This glibc tuning is not applied to all Python services because I
haven't done proper testings on them and at first glance they don't seem
to have such great improvements.

Related-bug: #1908805
Change-Id: Ic9030d01468b3189350f83b04a8d1d346c489d3c
(cherry picked from commit d5af514a)
(cherry picked from commit eb16f28ab2ee5ab8642bb404e1d0bcec75399f32)
(cherry picked from commit 0d223809)

... when installed from distribution.

This is mostly to fix Ironic's gate as their ecosystem is too
broad and complex to quickly remove libvirt-python from all
possible requirements.txt

More details inline.

See also: https://review.opendev.org/c/openstack/devstack/+/798514
aka f0bf2bdf

Change-Id: Ic44daf15e952bbe3c424984ffb2474261e68008f
(cherry picked from commit 8b8a4c75)

This reverts commit 7a3a7ce8 and
bcd0acf6 and part of
f1ed7c77 which all cap our pip
installs.

Given the pip ecosystem can often incorporate major changes, tracking
upstream at least generally gives us one problem at a time to solve
rather than trying to handle version jumps when LTS distros update.

The new dependency resolver included some changes that disallow
setting URL's like "file:///path/to/project#egg=project" in
constraints.  Apparently the fact it used to work was an accident of
the requires/constraints mechanism; it does make some sense as the URL
doesn't really have a version-number that the resolver can put in an
ordering graph.

The _setup_package_with_constraints_edit function comment highlights
what this is trying to do

 # Updates the constraints from REQUIREMENTS_DIR to reflect the
 # future installed state of this package. This ensures when we
 # install this package we get the from source version.

In other words; if constraints has "foo==1.2.3" and Zuul has checked
out "foo" for testing, we have to make sure pip doesn't choose version
1.2.3 from pypi.

It seems like removing the entry from upper-requirements.txt is the
important part; adding the URL path to the on-disk version was just
something that seemed to work at the time, but isn't really necessary.
We will install the package in question which will be the latest
version (from Zuul checkout) and without the package in
upper-requirements.txt nothing will try and downgrade it.

Therefore the solution proposed here is to remove the adding of the
URL parts.

This allows us to uncap pip and restore testing with the new
dependency resolver.

Closes-Bug: #1906322
Change-Id: Ib9ba52147199a9d6d0293182d5db50c4a567d677
(cherry picked from commit 6b9a5646)

These are no longer working, so drop'em.

Change-Id: I4e8311dd4e1eb7c9bc9533cebbd1a90dba801392

Without it segment plugin fails to connect with
placement api. Configure the placement section
if service is deployed.

Closes-Bug: #1973783
Change-Id: Ie7f37770a04f622735cf2263c601257669ab5064
(cherry picked from commit 92a34dbe)

This is necessary for more consistent behavior across multiple
distro versions. Apparently somewhere along the way, git started
looking at the current user's home directory instead of $HOME.

Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798

Change-Id: I941ef5ea90970a0901236afe81c551aaf24ac1d8
(cherry picked from commit 4baeb3b5)
(cherry picked from commit 9616d229)
(cherry picked from commit d86f23b1)

git commit [1] introduced a new behaviour to work around a CVE that
disallows any git operations in directories not owned by the current
user.

This may seem unrelated to installation, but it plays havoc with PBR,
which calls out to git to get to get revision history.  So if you are
"pip install"-ing from a source tree you don't own, the PBR git calls
in that tree now fail and the install blows up.

This plays havoc with our model.  Firstly, we checkout all code as
"stack" then install it globally with "sudo" (i.e. root) -- which
breaks.  We also have cases of essentially the opposite -- checkouts
we have installed as root, but then run tox in them as a regular user;
tox wants to install the source in its venv but now we have another
user conflict.

This uses the only available configuration option to avoid that by
globally setting the source directories we clone as safe.  This is an
encroachment of the global system for sure, but is about the only
switch available at the moment.  For discussion of other approaches,
see [2].

Related-Bug: https://bugs.launchpad.net/devstack/+bug/1968798

[1] https://github.com/git/git/commit/8959555cee7ec045958f9b6dd62e541affb7e7d9
[2] https://review.opendev.org/c/openstack/devstack/+/837636

Depends-On: https://review.opendev.org/c/openstack/devstack/+/837745
Change-Id: Ib9896a99b6d6c4d359ee412743ce30512b3c4fb7
(cherry picked from commit 676dcaf9)

This change enables [workarounds]libvirt_disable_apic when devstack is
deployed using the libvirt virt driver and qemu virt type in an effort
to avoid issues outlined in bug #1939108 caused by the older kernel
currently used in Cirros 0.5.2.

Depends-On: https://review.opendev.org/c/openstack/nova/+/805628
Closes-Bug: #1939108
Change-Id: Ibb6c34133bb1c95ef11cc59d9b12a0f65502c61b
(cherry picked from commit 1e86a25c)

The default CHAP algorithm for iscsid is md5, which is disallowed
under fips.  We will set the chap algorithm to "SHA3-256,SHA256",
which should work under all configurations.

For some reason, setting the CHAPAlgorithms as in c3b70513
breaks OpenEuler.  Making this conditional so that tests continue
to pass.

Change-Id: Iaa740ecfbb9173dd97e90485bad88225caedb523
(cherry picked from commit ac958698)

Since pip v22, python3.6 is not supported (the minimum version is
python3.7). This patch adds the reference for the pip3.6 URL to be
used instead of the default one.

Conflicts:
  tools/install_pip.sh

Closes-Bug: #1959600
Change-Id: Iab2c391d5388461fe9e9037cee81884ce8032e72
(cherry picked from commit a756f4b9)
(cherry picked from commit 13da39fc)

As added in notes for INSTALL_TEMPEST variable we need to set
this as False for stable branch so that devstack does not
install Tempest at system wide.

- https://github.com/openstack/devstack/blob/stable/xena/lib/tempest#L61

This should be done at the time when we cut the stable branch but
we forgot to do that for wallaby and xena. I have updated the QA
release wiki[1] for this to take care in future.

[1] https://wiki.openstack.org/wiki/QA/releases#Projects_with_only_Branches

Change-Id: I65304f66b3fe9c372fad8d4b521d0219833c1bf1

We still are seeing regular job failures because the pcp package fails
to install. Assume that we can still enable it on demand when someone
needs to debug specific job issues, let us just disable it by default.

Related-Bug: 1943184
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: I32ef8038e21c818623db9389588b3c6d3f98dcad
(cherry picked from commit 134205c1)

When deploying devstack in a stable branch, the master branch is
available locally only in a CI environment where Zuul prepares all
available branches. For a non-CI deployment we need to stick to using
the remote branch as was the case before [0].

While the situation on the master branch isn't really broken, we apply
the fix here anyway so that future stable branches are created in a
working state.

[0] I5d42ac6b54bf20804d7e5faa39d1289102318b64

Closes-Bug: #1956219
Signed-off-by: Dr. Jens Harbott <harbott@osism.tech>
Change-Id: Ib7719cb2d48b34db70f885e0afe77d904abba3b5
(cherry picked from commit 2ef4a4c8)

Currently upper-constraints.txt is not getting used
from in-review changes of requirements project and
leading to merge of broken requirements[1].

Use master branch to fetch constraints instead of
the remote branch.

[1] https://review.opendev.org/c/openstack/requirements/+/822575

Depends-On: https://review.opendev.org/c/openstack/requirements/+/823128
Change-Id: I5d42ac6b54bf20804d7e5faa39d1289102318b64
(cherry picked from commit 05e622ea)

The current initiator name embedded in our CI images is not unique at
present and can often cause failures during live migrations with
attached volumes. This change ensures the name is unique by running
iscsi-iname again and overwriting the existing name.

We could potentially do this during the image build process itself but
given that devstack systems are not supposed to be multi-purpose this
should be safe to do during the devstack run.

NOTE(lyarwood): Conflict due to
If2f74f146a166b9721540aaf3f1f9fce3030525c not being present on
stable/wallaby.

Conflicts:
    lib/nova

Closes-Bug: #1945983
Change-Id: I9ed26a17858df96c04be9ae52bf2e33e023869a5
(cherry picked from commit 714826d1)
(cherry picked from commit ee629cc7)

nova-ceph-multistore was failing on stable branch
as it was picking master version of base job instead
of stable one. There is issue in job variant in base
ceph job which is now fixed by depends-on.

Re-enable it in devstack gate.

Depends-On: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/811348
Depedns-On: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/811478
Change-Id: Ia7f53b41f2450fb0dd7743f7dc7024cd987feeb7

The apache mod_proxy documentation[0] says that trailing slashes need to
match for the ProxyPass statement. Since adding a slash to the redirected
url would break things that need to access endpoints like /identity
without anything added, we need to drop the trailing slash for the
target URL. See [1] for the discussion of the CVE fix that changed the
previous behavior.

[0] https://httpd.apache.org/docs/trunk/mod/mod_proxy.html#proxypass
[1] https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1945274

For stable/wallaby the nova-ceph-multistore job is currently broken,
drop it for now, it can be re-added when they got fixed.

Depends-On: https://review.opendev.org/c/openstack/devstack/+/811399
Change-Id: Ia6bae7419e5d9a9585f2316562a6c6bb968dcaa7

Ceph adds the osd pool default size option on ceph.conf via [1];
this means we don't need to specify the size of this pool if
the same value (same variable) is used (CEPH_REPLICAS).
This change is an attempt of removing the size setting, relying
on the implicit declaration of the value provided by ceph.conf.

[1] https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L425

Change-Id: I5fa2105ceb3b97a4e38926d76c1e4028f1108d4a
(cherry picked from commit 448db9ec)