=='''Last Update:''' May 22, 2022=Objective===

'''NoteNOTE:''' this project is not about running Amahi platform software on virtual servers, etc. For that there is a separate page on [[Virtualization]].

===Goals===

* run some of Operate internal build machines in a reliable, efficient way, so that we have consistent and updated builds/releases* have consistent Consistent and recent backups making things recoverable* run Execute some testing of Amahi apps more easily and efficiently* test Test new features in an isolated manner...

===Known Issues===* Image resizing does not work* Controller node is memory intensive ===Pending Actions===* Set up floating IP address range* Create automated VM backup routine ===Hardware==Configuration='''Dell Rack Server(F1DH):'''

* 8GB PC2-5300 RAM (8x1)* Two Gigabit Network InterfacesInterface

<br>'''Server #1 (CURRENT):'''* OpenStack release Yoga* Operating System: CentOS 8 Stream* 32GB PC2-5300 RAM (8x4)* Four Quick Swap Drive Bays** 1 - 1TB** 2 - 750GB (Backup)** 3 - EMPTY** 4 - EMPTY<br>'''Server #2 (BACKUP):'''* OpenStack release Wallaby* Operating System: CentOS 8 Stream* 24GB PC2-5300 RAM (4x2/8x2)* Quick Swap Drive Bays** 1 TB - 850GB (OS and BackupLVM)** 2 - EMPTY** 3 - 120GB SSD (VMsBackup)** 4 - EMPTY =Setup=The following guidance is based on the latest release of OpenStack (Yoga):<ul><li> Download [https://www.centos.org/download/ CentOS 8 Stream] minmal image and install following [https://linuxhint.com/install_centos8_netboot_iso/ Installing CentOS 8 using NetBoot ISO Image] tutorial (use LVM, but remove ''/home'' partition and redistribute space to ''/''). </li><li> [https://linuxconfig.org/rhel-8-configure-static-ip-address Set Static IP Address] and [https://techjourney.net/how-to-set-change-hostname-fqdn-fully-qualified-domain-name-in-rhel-centos-7/ configure Fully Qualified Domain Name], then reboot</li><li> Follow step 0 - 3 of the [https://www.rdoproject.org/install/packstack/ RDO Packstack Quickstart] for the OpenStack installation (ensure step 0 prerequisites are completed first).</li><li> At step 4, follow the [https://www.rdoproject.org/networking/neutron-with-existing-external-network/ Neutron with existing network guidance]:</li><ul><li> Use command: <code>packstack --allinone --provision-demo=n --cinder-volumes-size=100G --keystone-admin-passwd={set password} --os-neutron-ml2-mechanism-drivers=openvswitch --os-neutron-ml2-tenant-network-types=vxlan --os-neutron-ml2-type-drivers=vxlan,flat,vlan --os-neutron-l2-agent=openvswitch --os-neutron-ovs-bridge-mappings=extnet:br-ex --os-neutron-ovs-bridge-interfaces=br-ex:eth0 --nova-libvirt-virt-type=kvm</code> (replace {set password} and ''eth0'' with appropriate values)</li><li> Skip down to ''Now, create the external network with Neutron.'' and follow guidance.</li><li> Next move to ''Then create a router and set its gateway using the external network...'' and follow remaining guidance.</li></ul><li> Set up the floating IP addresses to the external network.</li><li> Create flavors named ''amahi.small'' of 10GB and ''amahi.medium'' of 15GB.<li> Add the following rules to ''default'' Security Group (Ingress, IPv4, Remote IP 0.0.0.0):<ul><li> ICMP / Any</li><li> TCP / 22</li><li> TCP / 80</li><li> TCP / 3000</li><li> TCP / 443 (<i>Optional</i>)</li></ul><li> Add users and private keys for SSH login</li><li> Disable SSH password and root login</li><li> Add users to web UI access</li><li> Ensure OS updates applied and reboot</li><li> Create a single instance with floating IP</li><li> Verify all is working as expected.</li></ul>  <b>NOTE:</b> A terminal script has been developed to automate the installation steps above for OpenStack after CentOS is installed. =Naming Convention=The following is recommended for standardization:* Instances: '''os-function''' (i.e. ''f24-bot'', ''f24-repo'', ''f24-dev'', etc)* Images: '''os-type''' (i.e. ''f24-cd'', ''f24-dvd'', etc)* Snapshot: '''os-function-ss#''' (i.e. ''f24-repo-ss1'', ''f24-bot-ss2'', etc)* Volumes: '''instance-vol''' (i.e. ''f24-repo-vol'', ''mirrormgr-vol'', ''dlmaster-vol'', etc) =Build Images=This will outline how to build OpenStack images using [https://www.proxmox.com/en/ Proxmox VE] (alternate method in development).* Log into Proxmox VE web UI* Create a VM or clone an existing one (<code><i>type</i>=qemu64</code>)** If creating a VM, install the OS** If using a clone, start the VM* Open a console window for the VM** Log in and as <code>root</code> do the following*** <code>dd if=/dev/zero of=/mytempfile bs=1M</code> (zero out any unused space)*** <code>rm -f /mytempfile</code>** Shutdown VM** Log into Proxmox VE via SSH and execute the following from command line*** Navigate to <code>/dev/pve</code>*** <code>qemu-img convert -O qcow2 original_image new_image.qcow2</code>*** Copy <code>new_image.qcow2</code> to a safe location for uploading into OpenStack*** Delete the VM from Proxmox VE web UI* Use WinSCP or similar program to copy the <code>new_image.qcow2</code> to client machine* Upload into OpenStack via the web UI  '''Reference:''' [https://gazelle.ihe.net/book/export/html/917 Transfer Proxmox VM] =Create Instance=This is a nice straightforward tutorial on [https://www.tecmint.com/create-deploy-and-launch-virtual-machines-in-openstack/ how to deploy and launch an instance (VM)]. Also refer to [https://docs.openstack.org/mitaka/user-guide/cli_nova_launch_instance_using_ISO_image.html launch instance from ISO] for an alternate method. =Backup='''Last Backup completed:'''* Instances - ''04 Jul 2021''* Volumes - ''04 Jul 2021''  '''NOTES:'''* Back up scripts have been created to synchronize instances, volumes, and snapshots to a secondary drive on demand.* Minimal of monthly back ups be done in case of catastrophic failure.* See [https://github.com/Nexenture/openstack-instances-backups OpenStack Instance Backups] to automate via terminal script (some modification is needed). =Tips===Command Line==[https://docs.openstack.org/python- Emptyopenstackclient/ussuri/ OpenStack Command Line Guide]==Network Issues==If the DNS server is changed or networking appears inoperable, check the following to ensure the DNS is correct:*''/etc/sysconfig/network-scripts/ifcfg-br-ex''* 4 ''/etc/resolve.conf''  Next restart neutron network services:<code>:service neutron-server restart<br />:service neutron-dhcp-agent restart<br />:service neutron-l3-agent restart<br />:service neutron-metadata-agent restart<br />:service neutron-openvswitch-agent restart</code> ==Volume Issues==When a volume becomes detached and/or shows in error, the state can be reset:<code>:source keystonerc_admin:cinder reset-state ''volume_id''</code>or use web UI. Also refer to [https://bugs.amahi.org/issues/2051 Amahi Bug #2051]. ==Instance Issues==When an instance '''Status''' reflects an ''error'' state, reset it by:<code>:source keystonerc_admin:nova reset-state --active INSTANCE_ID</code> '''NOTE:''' Should the '''Power State''' reflect ''no state'' or blank, it can be corrected by doing the above step, then starting the instance. ==Update/Reboot/Shutdown Process==Follow guidelines below to prevent corruption:* Shutdown/Disconnect** Stop all instances** Verify all instances stopped** Perform Update/Reboot/Shutdown<br>* Once system Started/Rebooted:** Start all needed instances** Verify all instances are operational ==Create Static IP for Instance==Setting a static IP for instances will ensure the internal IP address remains the same through it's life cycle. The floating IP address can be easily added afterwards.* As root user, execute <code>source keystonerc_admin</code>* Use the following to reserve the IP address:<code>:openstack port create --network private --fixed-ip subnet_id=private_subnet,ip_address=x.x.x.x static-ip-1</code>* Create the instance and boot via command-line vs the web UI:<code>:openstack server create --image IMAGE_NAME --flavor amahi.small --nic port-id=NETWORK_ID_FROM_ABOVE_COMMAND INSTANCE_NAME</code> :'''NOTE:''' If image name has spaces, enclose it in double quotes.Ref: [https://ask.openstack.org/en/question/30690/add-multiple-specific-ips-to-instance/ Add Multiple Specific IPs to Instance] ==Change Instance Flavor==As ''root'' user:<code>:source keystonerc_admin</code>See [https://docs.openstack.org/newton/user-guide/cli-change-the-size-of-your-server Change the size of your server] for command line guidance. ==Fedora Cloud Base Images==See [https://alt.fedoraproject.org/cloud/ Fedora Cloud Base Images] for guidance. == Migration ===== Images ===See [https://raymii.org/s/tutorials/Openstack_Glance_Image_Download.html OpenStack Glance Image Download Tutorial] for how to transfer images from one server to another. === Snapshots ===See [https://community.cisco.com/t5/cloud-blogs/how-to-snapshot-an-openstack-nova-instance-boot-it-download/ba-p/3661228 Snapshot an OpenStack instance, boot it, and download] for how to transfer snapshots from one instance to another. === Volumes ===See [http://blog.oddbit.com/2015/09/29/migrating-cinder-volumes-between-openstack- Emptyenvironments/ Migrating cinder volumes between OpenStack environments] for how to transfer volumes from one instance to another.

===Software=Fix Broken Web UI==* CentOS 7 x86_64 (Minimal)Add the following line to ''/etc/httpd/conf.d/15-horizon_vhost.conf'': WSGIApplicationGroup %{GLOBAL}* OpenStack Juno ReleaseThen restart the web server

===Setup===* Download and install [httpRef://isoredirect.centos.org/centos/7/isos/x86_64/ CentOS 7 x86_64 minimal] image* Configure FQDN (<code>/etc/hosts</code> and <code>/etc/hostname</code>)* Manually configure networking (set static IP address)* Add users and private keys for SSH login* Disable SSH password and root login* Enable EPEL Repo yum install epel-release:or rpm -Uvh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-2.noarch.rpm* Perform OS update yum -y update* Install OpenStack following [https://ask.openstack.redhat.comorg/Quickstart RDO Quickstart] instructions (run <code>packstack --allinone<en/code> as '''root''')* Configure network bridging (refer to [https:question/91352/openstack.redhat.com/Neutron_with_existing_external_network RDO Reference])** Set <code>CONFIG_PROVISION_ALL_IN_ONE_OVS_BRIDGE=n</code> in ''packstack-answersmitaka-20141028can-205455.txt''** Executed <code>packstack not-access-answer-file=packstack-answers-20141028-205455.txt<dashboard/code> (as '''root''')** Created ''/etc/sysconfig/network-scripts/ifcfg-br-ex''** Revised ''/etc/sysconfig/network-scripts/ifcfg-enp9s0f0''** Appended lines to ''/etc/neutron/plugin.ini''** Restarted the network service ** Removed router and subnet ('''ALL''' instances must be terminated to remove subnet)** Recreated subnet with IP address allocation range (set floating IP addresses)** Recreated router to match gateway* Configure DNS to OpenStack cannot access internet** Edit <code>/etc/neutron/dhcp_agent.ini</code> and uncomment the line below:<pre># Comma-separated list of DNS servers which will be used by dnsmasq# as forwarders.# dnsmasq_dns_servers = </pre>** Add <code>8.8.8.8,8.8.4.4</code> after equal (=)** Restart system as I could not determine what service to restartDashboard]

===Notes==Known Issues=* the floating IPs situation may not work on non-externally routed IPsCannot use multiple SSH keys via Dashboard ([https://bugs. this may be why they set up a 179. "public" network by detafult in the RDO setup. i deleted that network * the external network needs to be "flagged" as external. this cannot be done with the UI, but i am told the juno release has a feature where attribute editing. so that the external attribute can be set to Yes. once that is done, MAYBE the system allows floating IPs in that network even if the IP range is not externally routable* basically understand what it takes to get an image created, seasoned, and how we need to maintain these over long periods. i think the main workhorse is qcow2 tools.* these images are like "snapshots" in some way, but a snapshot is frozen and cannot be tweaked.* long term we want to make images like this for testing, e.g. and amahi 7 image that is bootable and it's plain instal. another example is a fully up to date amahi 7 image, etc.* so they are alive in that these images are frozen in time, but one takes a copy and can then evolve it into a new version of the imagelaunchpad.net/nova/+bug/1499751 OpenStack Bug #1499751])<br />