Changes

From Amahi Wiki
Jump to: navigation, search
{{WorkInProgress}}
'''Last Update:''' May 22, 2022
=Objective=
This is a project to update and modernize the infrastructure that keeps the Amahi web sites and services running.
=Goals=
We have multiple goals:
* run some of Operate internal build machines in a reliable, efficient way, so that we have consistent and updated builds/releases* have consistent Consistent and recent backups making things recoverable* run Execute some testing of Amahi apps more easily and efficiently* test Test new features in an isolated manner
=Known IssuesConfiguration=* Cannot use multiple SSH keys via Dashboard '''Dell Server ([httpsF1DH)://bugs.launchpad.net/nova/+bug/1499751 OpenStack Bug #1499751]) =Hardware=Dell Rack Server'''
* Dual Xeon E5450 3.0 GHz Processors
* 32GB PC2-5300 RAM (8x4)* Two Gigabit Network InterfacesInterface
* KVM Network Interface
* RAID Controller
<br>'''Server #1 (CURRENT):'''* OpenStack release Yoga* Operating System: CentOS 8 Stream* 32GB PC2-5300 RAM (8x4)* Four Quick Swap Drive Bays** 1 - 850GB (OS and Backup)1TB** 2 - 2 TB 750GB (Images and Backup)** 3 - 120GB SSD EMPTY** 4 - EMPTY<br>'''Server #2 (VMsBACKUP):'''* OpenStack release Wallaby*Operating System: CentOS 8 Stream* 4 24GB PC2- Empty5300 RAM (4x2/8x2)* Quick Swap Drive Bays=Software=** 1 - 850GB (LVM)* CentOS 7.* 2 x86_64 - EMPTY** 3 - 120GB SSD (MinimalBackup)* OpenStack Mitaka Release* 4 - EMPTY
=Setup=
The following guidance is based on the latest release of OpenStack (Yoga):
<ul>
<li> Download [httphttps://mirrorswww.catcentos.pdxorg/download/ CentOS 8 Stream] minmal image and install following [https://linuxhint.educom/install_centos8_netboot_iso/ Installing CentOS 8 using NetBoot ISO Image] tutorial (use LVM, but remove ''/centoshome'' partition and redistribute space to ''/''). </7li><li> [https:/isos/x86_64linuxconfig.org/CentOSrhel-8-7configure-x86_64static-Minimalip-1511address Set Static IP Address] and [https://techjourney.iso CentOS net/how-to-set-change-hostname-fqdn-fully-qualified-domain-name-in-rhel-centos-7/ configure Fully Qualified Domain Name], then reboot</li><li> Follow step 0 - 3 of the [https://www.rdoproject.2 x86_64 minimalorg/install/packstack/ RDO Packstack Quickstart] image and install following for the OpenStack installation (ensure step 0 prerequisites are completed first).</li><li> At step 4, follow the [httphttps://www.ifrdoproject.org/networking/neutron-notwith-trueexisting-thenexternal-false.comnetwork/2014Neutron with existing network guidance]:</centosli><ul><li> Use command: <code>packstack --allinone --provision-demo=n --cinder-volumes-size=100G --keystone-admin-passwd={set password} --os-neutron-ml2-mechanism-drivers=openvswitch --os-neutron-ml2-tenant-network-types=vxlan --os-neutron-ml2-type-drivers=vxlan,flat,vlan --os-neutron-l2-agent=openvswitch --os-neutron-ovs-bridge-mappings=extnet:br-ex --os-neutron-ovs-bridge-interfaces=br-ex:eth0 --nova-7libvirt-netinstallvirt-guidetype=kvm</code> (replace {set password} and ''eth0'' with appropriate values)</li><li> Skip down to ''Now, create the external network with Neutron.'' and follow guidance.</ CentOS 7li><li> Next move to ''Then create a router and set its gateway using the external network...'' and follow remaining guidance.2 Netinstall Guide] tutorial</li></ul><li> Set up the floating IP addresses to the external network. </li><li> Configure FQDN Create flavors named ''amahi.small'' of 10GB and ''amahi.medium'' of 15GB.<li> Add the following rules to ''default'' Security Group (Ingress, IPv4, Remote IP 0.0.0.0):<ul><codeli> ICMP / Any</li><li>TCP /etc22</li><li> TCP /hosts80</codeli> and <codeli>TCP /etc3000</hostnameli><li> TCP / 443 (<i>Optional</codei>)</li></ul>
<li> Add users and private keys for SSH login</li>
<li> Disable SSH password and root login</li>
<li> Follow [https://openstack.redhat.com/Quickstart RDO Quickstart] for the OpenStack installation guidance.Add users to web UI access</li><li> At the <code>packstack --allinone</code> step, follow the [https://www.rdoproject.org/networking/neutron-with-existing-external-network/ Neutron with existing network guidance] instead.Ensure OS updates applied and reboot</li><li> Refer to [https://www.rdoproject.org/networking/floating-ip-range/ floating IP range] for setting up the Create a single instance with floating IP addresses to the external network.</li><li> Extend cinder-volumes past 20GB to allow for creating additional volumes to attach to instancesVerify all is working as expected.</li></ul>  <lib> Followed the [httpsNOTE://udaraliyanage.wordpress.com/2014/05/23/openstack-increase-volume-capacity OpenStack Increase Volume Capacity] tutorial</lib><li> Created ''/usr/bin/ext-cinder-vol'' A terminal script:</li><code>#!/bin/bash<br />/usr/sbin/losetup -f /var/lib/cinder/cinder-volumes-ext<br />/usr/bin/systemctl restart openstack-cinder-volume<br />/usr/bin/systemctl restart openstack-cinder-api<br />/usr/bin/systemctl restart openstack-cinder-scheduler</code><li> Add has been developed to root crontab:</li><code>@reboot /usr/bin/ext-cinder-vol</code></li><li> Results in 130Gb additional space automate the installation steps above for volumes.</li><li> Total volume space available OpenStack after CentOS is now 150GBinstalled.</li></ul></ul>
=Naming Convention=
The following is recommended for standardization:
* Instances: '''os-function''' (i.e. ''f24-bot'', ''f24-repo'', ''f24-dev'', etc)
* Images: '''os-type''' (i.e. ''f24-cd'', ''f24-dvd'', etc)
* Snapshot: '''os-function-ss#''' (i.e. ''f24-repo-ss1'', ''f24-bot-ss2'', etc)
* Volumes: '''instance-vol''' (i.e. ''f24-repo-vol'', ''mirrormgr-vol'', ''dl-masterdlmaster-vol'', etc)
=Build Images=
This will outline how to build OpenStack images using [https://www.proxmox.com/en/ Proxmox VE] (alternate method in development).
* Log into Proxmox VE web UI
* Create a VM or clone an existing one(<code><i>type</i>=qemu64</code>)
** If creating a VM, install the OS
** If using a clone, start the VM
* Open a console window for the VM
** Log in and as <code>root </code> do the following
*** <code>dd if=/dev/zero of=/mytempfile bs=1M</code> (zero out any unused space)
*** <code>rm -f /mytempfile</code>
** Shutdown VM
** Log into Proxmox VE via SSH and execute the following from command line
*** Navigate to <code>/vardev/lib/vz/images/###pve</code> (number of VM)*** <code>mv original_image.qcow2 original_image.qcow2_backup</code> (rename original image)*** <code>qemu-img convert -O qcow2 original_image.qcow2_backup original_imagenew_image.qcow2</code>*** Copy new <code>new_image.qcow2</code> image to a safe location for uploading into OpenStack*** Remove <code>.backup</code> file
*** Delete the VM from Proxmox VE web UI
* Use WINScp WinSCP or similar program to copy the <code>new_image.qcow2</code> image to client machine
* Upload into OpenStack via the web UI
'''REFReference:''' [httphttps://wwwgazelle.jamescoyleihe.net/how-tobook/323-reclaim-disk-space-from-a-sparse-image-file-qcow2-vmdk Reclaim disk space from .qcow2 or .vmdk imageexport/html/917 Transfer Proxmox VM]
=Create Instance=
This is a nice straightforward tutorial on [https://communitywww.hpcloudtecmint.com/article/creatingcreate-deploy-and-launch-virtual-yourmachines-firstin-openstack/ how to deploy and launch an instance(VM)]. Also refer to [https://docs.openstack.org/mitaka/user-0 Creating an guide/cli_nova_launch_instance_using_ISO_image.html launch instancefrom ISO]for an alternate method.
=Backup=
'''Last Backup completed:''' * Instances - ''04 Jul 2021''25 Aug 2016* Volumes - ''04 Jul 2021''  <br />'''NOTES:'''
* Back up scripts have been created to synchronize instances, volumes, and snapshots to a secondary drive on demand.
* Recommend Minimal of monthly back ups be done in case of catastrophic failure.* See [https://github.com/Nexenture/openstack-instances-backups OpenStack Instance Backups] to automate via terminal script (some modification is needed).
=Tips=
==Command Line==
[https://docs.openstack.org/python-openstackclient/ussuri/ OpenStack Command Line Guide]
==Network Issues==
If the DNS server is changed or networking appears inoperable, check the following to ensure the DNS is correct:
* ''/etc/sysconfig/network-scripts/ifcfg-br-ex''
* ''/etc/resolve.conf''
Next restart neutron network services:
<blockquote><code>:service neutron-server restart<br />:service neutron-dhcp-agent restart<br />:service neutron-l3-agent restart<br />:service neutron-metadata-agent restart<br />:service neutron-openvswitch-agent restart</code></blockquote>
==Volume Issues==
When a volume becomes detached and/or shows in error, the state can be reset:
:<code>:source keystonerc_admin</code>:<code>cinder reset-state ''volume_id''</code>
or use web UI.
Also refer to [https://bugs.amahi.org/issues/2051 Amahi Bug #2051].
 
==Instance Issues==
When an instance '''Status''' reflects an ''error'' state, reset it by:
<code>
:source keystonerc_admin
:nova reset-state --active INSTANCE_ID
</code>
 
'''NOTE:''' Should the '''Power State''' reflect ''no state'' or blank, it can be corrected by doing the above step, then starting the instance.
==Update/Reboot/Shutdown Process==
Follow guidelines below to prevent corruption:
* Shutdown/Disconnect
** Stop all instances via SSH** Detach volumes from instances** Verify all volumes detached and instances stopped** Perform Update/Reboot/Shutdown<br>* Once system rebootedStarted/Rebooted:** Verify cinder_volumes (LVM) is operational** Reattach volumes to instances
** Start all needed instances
** Verify all instances are operational
* As root user, execute <code>source keystonerc_admin</code>
* Use the following to reserve the IP address:
:<code>neutron :openstack portcreate --create internal network private --fixed-ip subnet_id=internal_subnetprivate_subnet,ip_address=x.x.x.xstatic-ip-1</code>
* Create the instance and boot via command-line vs the web UI:
:<code>nova boot :openstack server create --image NAME_OF_IMAGE IMAGE_NAME --flavor amahi.small --nic port-id=ID_FOR NETWORK_FROM_ABOVE_COMMAND_RESULTS VM_NAMENETWORK_ID_FROM_ABOVE_COMMAND INSTANCE_NAME</code>
:'''NOTE:''' If image name has spaces, enclose it in double quotes.
Ref: [https://ask.openstack.org/en/question/30690/add-multiple-specific-ips-to-instance/ Add Multiple Specific IPs to Instance]
==Fedora Cloud ImagesChange Instance Flavor==As ''root'' user:<code>:source keystonerc_admin</code>See [https://wwwdocs.rdoprojectopenstack.org/Running_an_instance#Step_5:_Launch_the_instance Launch Fedora Cloud imagesnewton/user-guide/cli-change-the-size-of-your-server Change the size of your server] for command line guidance.
==MiscellaneousFedora Cloud Base Images==Refer See [https://alt.fedoraproject.org/cloud/ Fedora Cloud Base Images] for guidance. == Migration ===== Images ===See [https://raymii.org/s/tutorials/Openstack_Glance_Image_Download.html OpenStack Glance Image Download Tutorial] for how to transfer images from one server to another. === Snapshots ===See [https://bugscommunity.amahicisco.orgcom/t5/cloud-blogs/how-to-snapshot-an-openstack-nova-instance-boot-it-download/issuesba-p/2050 Amahi Bug #20503661228 Snapshot an OpenStack instance, boot it, and download] for some how to transfer snapshots from one instance to another. === Volumes ===See [http://blog.oddbit.com/2015/09/29/migrating-cinder-volumes-between-openstack-environments/ Migrating cinder volumes between OpenStack Commandenvironments] for how to transfer volumes from one instance to another. ==Fix Broken Web UI==Add the following line to ''/etc/httpd/conf.d/15-Line syntaxhorizon_vhost.conf'': WSGIApplicationGroup %{GLOBAL}Then restart the web server Ref: [https://ask.openstack.org/en/question/91352/openstack-mitaka-can-not-access-dashboard/ OpenStack cannot access Dashboard] =Known Issues=Cannot use multiple SSH keys via Dashboard ([https://bugs.launchpad.net/nova/+bug/1499751 OpenStack Bug #1499751])<br>
12,424

edits