| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Automatic merge from submit-queue
resolve #5428: python-dbus not found
`python-dbus` is not available in centos standard repos, but:
> It appears python-dbus is just a reference to dbus-python
and `dbus-python` is.
|
| | | |
|
| |\ \
| | |
| | |
| | |
| | | |
Automatic merge from submit-queue
Use 3.7 RPM repo
|
| | | | |
|
| |\ \ \
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Automatic merge from submit-queue
crio: skip installation on lbs and nfs nodes
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1494461
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1494461
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
|
| |\ \ \ \
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
Automatic merge from submit-queue
Remove override default.py callback plugin
The functionality of this plugin has been added to Ansible as the [debug.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/debug.py) callback plugin. The Ansible default plugin has added a significant amount of functionality as well as updates to the CallbackBase class. When developing new plugins for OpenShift-Ansible this override can result in [unexpected](https://github.com/ansible/ansible/issues/27151) behavior.
This plugin was originally added in #1861.
|
| | | | | |
| | | | |
| | | | |
| | | | |
| | | | | |
The functionality of this plugin has been added to Ansible as the
debug.py callback plugin.
|
| |\ \ \ \ \
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
Automatic merge from submit-queue
Fixed AnsibleUnsafeText by converting to int
Related to https://github.com/openshift/openshift-ansible/pull/5067
I've tested with custom values and it failed. This fixes the issue in my tests.
```
[cloud-user@bastion ~]$ grep -E 'osm|portal' /etc/ansible/hosts
osm_default_node_selector="role=app"
osm_use_cockpit=true
osm_cluster_network_cidr=10.130.0.0/14
osm_host_subnet_length=8
openshift_portal_net=10.111.0.0/16
```
After the installation:
```
[root@master-0 ~]# alias oetcdctl='etcdctl --cert-file=/etc/etcd/peer.crt --key-file=/etc/etcd/peer.key --ca-file=/etc/etcd/ca.crt --peers="https://master-0.edu.flannel.com:2379,https://master-1.edu.flannel.com:2379,https://master-2.edu.flannel.com:2379"'
[root@master-0 ~]# oetcdctl get /openshift.com/network/config
{
"Network": "10.130.0.0/14",
"SubnetLen": 24,
"Backend": {
"Type": "host-gw"
}
}
```
But, the subnets assigned to the nodes are on different subnet:
```
[root@master-0 ~]# oetcdctl ls /openshift.com/network/subnets
/openshift.com/network/subnets/10.128.83.0-24
/openshift.com/network/subnets/10.128.18.0-24
/openshift.com/network/subnets/10.128.77.0-24
/openshift.com/network/subnets/10.128.101.0-24
/openshift.com/network/subnets/10.128.20.0-24
/openshift.com/network/subnets/10.128.92.0-24
/openshift.com/network/subnets/10.128.58.0-24
/openshift.com/network/subnets/10.128.48.0-24
```
|
| | | |/ / /
| |/| | | |
|
| |\ \ \ \ \
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
Automatic merge from submit-queue
Improve CA redeploy restart logic
Expired etcd certificates require special casing around restarts in the certificate redeploy playbooks. When etcd certificates are expired we can't restart masters or nodes. We also can't simply restart etcd because peers also had expired certificates so we must start/stop etcd when we detect expired etcd certificates.
`openshift-ca.yml`:
* No longer restart master services when etcd certificates were previously expired.
* No longer restart node services when master or etcd certificates were previously expired.
`etcd-ca.yml`:
* No longer restart master services when etcd certificates were previously expired.
Tested using [gen_expired_tls.sh](https://gist.github.com/abutcher/bdd20b9d582675d89fb22658689c49e4) on one of my master/etcd hosts to ensure that restart logic changes caused us to skip the right restarts and do a full start/stop of etcd in the `redeploy-certificates.yml` and `redeploy-etcd-certificates.yml` playbooks.
Note: When this happens with a cluster and you want to replace all certificates, you can run: `redeploy-etcd-ca.yml`, `redeploy-openshift-ca.yml` (which will both skip restarts) and then run `redeploy-certificates.yml` which will now be able to full/stop start etcd.
|
| | | | | | | |
|
| | | |/ / /
| |/| | |
| | | | |
| | | | | |
had previously expired certificates.
|
| |\ \ \ \ \
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | |
| | | | | | |
Automatic merge from submit-queue
Set master facts prior to adding new etcd client urls to master config.
`openshift_master` role dependencies were moved out of the role in https://github.com/openshift/openshift-ansible/pull/5392 so we need to call `openshift_master_facts` prior to patching the master config in etcd scaleup.
https://bugzilla.redhat.com/show_bug.cgi?id=1490304
|
| | | | | | | |
|
| |\ \ \ \ \ \
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
Automatic merge from submit-queue
Fix registry_auth logic for upgrades
Currently, the logic for registry authentication is
not implemented correctly to account for upgrades of
containerized hosts.
Additionally, the logic to account for multiple runs
of openshift-ansible might cause registry authentication
credentials to not be mounted inside of containerized hosts.
This commit adds the necessary logic to ensure containerized
hosts retain registry credentials.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1494470
|
| | | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
Currently, the logic for registry authentication is
not implemented correctly to account for upgrades of
containerized hosts.
Additionally, the logic to account for multiple runs
of openshift-ansible might cause registry authentication
credentials to not be mounted inside of containerized hosts.
This commit adds the necessary logic to ensure containerized
hosts retain registry credentials.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1494470
|
| |\ \ \ \ \ \ \
| |_|_|_|_|_|/
|/| | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | | |
Automatic merge from submit-queue
Consolidate etcd migrate role
The PR is based on top of https://github.com/openshift/openshift-ansible/pull/5371 and https://github.com/openshift/openshift-ansible/pull/5451. Once both PRs are merged, I will rebase.
|
| | | | | | | | |
|
| |\ \ \ \ \ \ \
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
Automatic merge from submit-queue
Move some pre-checks before excluders are disabled
Some pre-checks needs an OCP version which is detected by a set of tasks that need the excluders to be disabled. So at the best I can move some pre-checks before the excluders are disabled. However, there will be still some checks that can fail with excluders updated to the newer version.
Bug: 1484304
|
| | | | | | | | | |
|
| |\ \ \ \ \ \ \ \
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
Automatic merge from submit-queue
move (and rename) get_dns_ip filter into openshift_node_facts
Due to recent changes to filter_plugins/openshift_node.py, when trying to run a current version of the openshift_node_facts role on a system with an older version of the openshift-ansible-filter-plugins RPM, the new role will fail.
```
File "/usr/share/ansible_plugins/filter_plugins/openshift_node.py", line 30, in get_dns_ip
if bool(hostvars['openshift']['common']['use_dnsmasq']):
KeyError: 'use_dnsmasq'
```
It's not possible for us to have our current RPM version (presently openshift-ansible-filter-plugins-3.5.120-1.git.0.c60f69a.el7.noarch) and a newer RPM installed and run both current openshift-ansible code and older checked-out/vendored releases (for older clusters/releases).
Since only the openshift_node_facts role uses the get_dns_ip filter, move the functionality into a role-specific filter plugin.
In addition, rename the filter plugin to 'node_get_dns_ip' because Ansible is preffering the get_dns_ip from the RPM (ie /usr/share/ansible_plugins/filter_plugins/openshift_node.py) over the role-specific filter plugin of the same name. Ansile prefers the filter plugins in /usr/share/ansible_plugins/filter_plugins/* even when /etc/ansible/ansible.cfg is set to something like:
filter_plugins = filter_plugins:/usr/share/ansible_plugins/filter_plugins:filter_plugins <--- yes, 'filter_plugins' before and after /usr/share/ansible_plugins/filter_plugins (ansible 2.3.1.0) (perhaps this is because the /usr/share/ansible/plugins/filter symlink to /usr/share/ansible_plugins/filter_plugins takes precedence over everything???)
Renaming the filter plugin function ensures that versions of the openshift_node_facts role that depend on the old implementation can continue to use it through what the older RPM provides, and the new role can use it's role-specific filter plugin without any variable namespace collisions.
Lastly, remove filter_plugins/openshift_node.py since it is now self-contained in roles/openshift_node_facts.
https://bugzilla.redhat.com/show_bug.cgi?id=1494312
|
| | | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
Due to a combination of unexpected behavior when resolving filter plugins in Ansible and recent changes to filter_plugins/openshift_node.py, when trying to run a current version of the openshift_node_facts role on a system with an older version of the openshift-ansible-filter-plugins RPM, the role will fail.
File "/usr/share/ansible_plugins/filter_plugins/openshift_node.py", line 30, in get_dns_ip
if bool(hostvars['openshift']['common']['use_dnsmasq']):
KeyError: 'use_dnsmasq'
It's not possible for us to have our current RPM version (presently openshift-ansible-filter-plugins-3.5.120-1.git.0.c60f69a.el7.noarch) and a newer RPM installed and run both current openshift-ansible code and older checked-out/vendored releases (for older clusters/releases).
Since only the openshift_node_facts role uses the get_dns_ip filter, move the functionality into a role-specific filter plugin.
In addition, rename the filter plugin to 'node_get_dns_ip' because Ansible is prefering the get_dns_ip from the RPM (ie /usr/share/ansible_plugins/filter_plugins/openshift_node.py) over the role-specific filter plugin of the same name. Ansile prefers the filter plugins in /usr/share/ansible_plugins/filter_plugins/* even when /etc/ansible/ansible.cfg is set to something like:
filter_plugins = filter_plugins:/usr/share/ansible_plugins/filter_plugins:filter_plugins <--- yes, 'filter_plugins' before and after /usr/share/ansible_plugins/filter_plugins (ansible 2.3.1.0)
Renaming the filter plugin function ensures that roles that depend on the old implementation can continue to use it through what the older RPM provides, and the new role can use it's role-specific filter plugin without any variable namespace colisions.
Lastly, remove filter_plugins/openshift_node.py since it is now self-contained in roles/openshift_node_facts.
|
| |\ \ \ \ \ \ \ \ \
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | | |
Automatic merge from submit-queue
health checks: add diagnostics check
Adds a health check that runs `oc adm diagnostics` with each individual diagnostic.
Also, moved `is_first_master` method into superclass for reuse. And look at `oo_first_master` and `ansible_host` instead of `masters` and `ansible_ssh_host`.
|
| | | |_|_|_|_|_|_|/
| |/| | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
Also, moved is_first_master method into superclass for reuse. And look
at oo_first_master and ansible_host instead of masters and
ansible_ssh_host.
|
| |\ \ \ \ \ \ \ \ \
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | | |
Automatic merge from submit-queue
Ensure that hostname is lowercase
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1396350
|
| | | |_|_|_|_|_|_|/
| |/| | | | | | |
| | | | | | | | |
| | | | | | | | | |
Fixes Bug 1396350
|
| |\ \ \ \ \ \ \ \ \
| | | | | | | | | |
| | | | | | | | | | |
Updating default behavior for installing metrics and logging. Separat…
|
| | | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | | |
uninstall to own variable
|
| |\ \ \ \ \ \ \ \ \ \
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | |
| | | | | | | | | | | |
Automatic merge from submit-queue
Remove unused openshift_hosted_logging role
This role has not been used for several releases.
It is not tested by an checks, and no meaningful
updates have been made to this role in several months.
|
| | | |_|_|_|_|_|_|_|/
| |/| | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | | |
This role has not been used for several releases.
It is not tested by an checks, and no meaningful
updates have been made to this role in several months.
|
| |\ \ \ \ \ \ \ \ \ \
| | | | | | | | | | |
| | | | | | | | | | | |
Add python3-PyYAML for Fedora installs
|
| | |/ / / / / / / / /
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | | |
Fedora installs currently fail because nodes require
the usage of python3. python3-PyYAML must be installed
in order to support usage of python3.
This commit adds the rpm python3-PyYAML to fedora
nodes.
Fixes: https://github.com/openshift/openshift-ansible/issues/5469
|
| |\ \ \ \ \ \ \ \ \ \
| |_|/ / / / / / / /
|/| | | | | | | | | |
Add booleans to prevent unwanted install of nuage roles.
|
| |/ / / / / / / / /
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
Recently, some role dependencies were removed from meta
depends into playbooks. Particularly, the nuage role
requires several roles. Currently, only the nuage_master
role requires openshift_use_nuage to be true.
This commit requires the other nuage roles to reference
the variable openshift_use_nuage before install.
|
| |\ \ \ \ \ \ \ \ \
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | | |
Automatic merge from submit-queue
Creating structure to warn for use of deprecated variables and set th…
…em in a single location before they are no longer honored
Implementation of https://trello.com/c/XKOGHSkP/460-3-deprecate-openshifthostedloggingmetrics-installer-variables
Tried to implement this in a way such that we can easily and cleanly add/remove variables in the future for any role we choose.
Addresses:
https://bugzilla.redhat.com/show_bug.cgi?id=1449812
https://bugzilla.redhat.com/show_bug.cgi?id=1464349
|
| | | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | | |
single location before they are no longer honored
|
| |\ \ \ \ \ \ \ \ \ \
| |_|_|_|_|_|_|_|_|/
|/| | | | | | | | | |
Remove debugging statements and pause module
|
| |/ / / / / / / / /
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
How on earth did the bot merge this? The upgrade test should've stalled
indefinitely.
|
| |\ \ \ \ \ \ \ \ \
| |_|_|_|_|_|_|_|/
|/| | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
Automatic merge from submit-queue
cri-o: Allow full image override
``openshift_crio_systemcontainer_image_registry_override`` has been replaced
with ``openshift_crio_systemcontainer_image_override``. The difference is
``openshift_crio_systemcontainer_image_override`` takes a full image path
including the tag.
Example:
```
openshift_crio_systemcontainer_image_override=gscrivano/cri-o-centos:latest
```
|
| | | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
openshift_crio_systemcontainer_image_registry_override has been replaced
with openshift_crio_systemcontainer_image_override. The difference is
openshift_crio_systemcontainer_image_override takes a full image path
including the tag.
Example:
openshift_crio_systemcontainer_image_override=gscrivano/cri-o-centos:latest
|
| |\ \ \ \ \ \ \ \ \
| |_|_|_|_|_|_|_|/
|/| | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
Automatic merge from submit-queue
Default openshift_pkg_version to full version-release during upgrades
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1490677
The versioning scheme for 3.7 pre-releases has changed and now all
versions are 3.7.0 and the release is incremented on builds, ie:
3.7.0-0.124.0 upgraded to 3.7.0-0.125.0. If we know we're an upgrade and
they haven't requested a specific package version defer the defaulting
of openshift_pkg_version until the upgrade playbooks and there set it to
the available version including the release.
|
| | | |/ / / / / /
| |/| | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | |
| | | | | | | | |
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1490677
The versioning scheme for 3.7 pre-releases has changed and now all
versions are 3.7.0 and the release is incremented on builds, ie:
3.7.0-0.124.0 upgraded to 3.7.0-0.125.0. If we know we're an upgrade and
they haven't requested a specific package version defer the defaulting
of openshift_pkg_version until the upgrade playbooks and there set it to
the available version including the release.
|
| |\ \ \ \ \ \ \ \
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
Automatic merge from submit-queue
Returning actual results of yedit query. Empty list was returning empty dict.
This is a simple change but since code generation happens it affects lots of files.
The only change that is happening here is the following:
```
- rval = yamlfile.get(params['key']) or {} 
+ rval = yamlfile.get(params['key'])
```
When something returned as 0, [], or None the query would return {}. This was unintended.
|
| | | |_|_|_|/ / /
| |/| | | | | | |
|
| |\ \ \ \ \ \ \ \
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
Automatic merge from submit-queue
Ensure valid search on resolv.conf
On cluster without internal name resolution there will be no `search XXX` on /etc/resolv.conf at all, thus this script will fail to add an entry for `cluster.local`.
Forward ports #5398
|
| | | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
On cluster without internal name resolution there will be no `search XXX` on /etc/resolv.conf at all, thus this script will fail to add an entry for `cluster.local`.
|
| |\ \ \ \ \ \ \ \ \
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | | |
Automatic merge from submit-queue
Creating initial tsb role to consume and apply templates provided for…
… tsb
cc: @deads2k @sdodson
Addresses:
https://bugzilla.redhat.com/show_bug.cgi?id=1486623
https://bugzilla.redhat.com/show_bug.cgi?id=1470623
https://bugzilla.redhat.com/show_bug.cgi?id=1491626
|
| | | |/ / / / / / /
| |/| | | | | | | |
|
| |\ \ \ \ \ \ \ \ \
| |_|_|_|_|/ / / /
|/| | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | |
| | | | | | | | | |
Automatic merge from submit-queue
consolidate etcd certs roles
This is a starter for consolidation of all etcd like roles into a single `etcd` action-based role. I have intentionally started with the simplest one to demonstrate the steps needed to make it so and to make the review easy enough for everyone.
|
