Difference between revisions of "Infrastructure Virtualization Project"
(165 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
{{WorkInProgress}} | {{WorkInProgress}} | ||
+ | '''Last Update:''' May 22, 2022 | ||
=Objective= | =Objective= | ||
This is a project to update and modernize the infrastructure that keeps the Amahi web sites and services running. | This is a project to update and modernize the infrastructure that keeps the Amahi web sites and services running. | ||
Line 9: | Line 10: | ||
=Goals= | =Goals= | ||
We have multiple goals: | We have multiple goals: | ||
− | * | + | * Operate internal build machines in a reliable, efficient way, so that we have consistent and updated builds/releases |
− | * | + | * Consistent and recent backups making things recoverable |
− | * | + | * Execute some testing of Amahi apps more easily and efficiently |
− | * | + | * Test new features in an isolated manner |
− | = | + | =Configuration= |
− | + | '''Dell Server (F1DH):''' | |
− | |||
− | |||
− | |||
* Dual Xeon E5450 3.0 GHz Processors | * Dual Xeon E5450 3.0 GHz Processors | ||
− | * | + | * Gigabit Network Interface |
− | |||
* KVM Network Interface | * KVM Network Interface | ||
* RAID Controller | * RAID Controller | ||
− | * | + | <br> |
− | ** 1 - | + | '''Server #1 (CURRENT):''' |
− | ** 2 - | + | * OpenStack release Yoga |
− | ** 3 - | + | * Operating System: CentOS 8 Stream |
− | ** | + | * 32GB PC2-5300 RAM (8x4) |
− | + | * Quick Swap Drive Bays | |
− | + | ** 1 - 1TB | |
− | * | + | ** 2 - 750GB (Backup) |
− | * | + | ** 3 - EMPTY |
+ | ** 4 - EMPTY | ||
+ | <br> | ||
+ | '''Server #2 (BACKUP):''' | ||
+ | * OpenStack release Wallaby | ||
+ | * Operating System: CentOS 8 Stream | ||
+ | * 24GB PC2-5300 RAM (4x2/8x2) | ||
+ | * Quick Swap Drive Bays | ||
+ | ** 1 - 850GB (LVM) | ||
+ | ** 2 - EMPTY | ||
+ | ** 3 - 120GB SSD (Backup) | ||
+ | ** 4 - EMPTY | ||
=Setup= | =Setup= | ||
+ | The following guidance is based on the latest release of OpenStack (Yoga): | ||
<ul> | <ul> | ||
− | <li> Download [ | + | <li> Download [https://www.centos.org/download/ CentOS 8 Stream] minmal image and install following [https://linuxhint.com/install_centos8_netboot_iso/ Installing CentOS 8 using NetBoot ISO Image] tutorial (use LVM, but remove ''/home'' partition and redistribute space to ''/''). </li> |
− | <li> | + | <li> [https://linuxconfig.org/rhel-8-configure-static-ip-address Set Static IP Address] and [https://techjourney.net/how-to-set-change-hostname-fqdn-fully-qualified-domain-name-in-rhel-centos-7/ configure Fully Qualified Domain Name], then reboot</li> |
+ | <li> Follow step 0 - 3 of the [https://www.rdoproject.org/install/packstack/ RDO Packstack Quickstart] for the OpenStack installation (ensure step 0 prerequisites are completed first).</li> | ||
+ | <li> At step 4, follow the [https://www.rdoproject.org/networking/neutron-with-existing-external-network/ Neutron with existing network guidance]:</li> | ||
+ | <ul> | ||
+ | <li> Use command: <code>packstack --allinone --provision-demo=n --cinder-volumes-size=100G --keystone-admin-passwd={set password} --os-neutron-ml2-mechanism-drivers=openvswitch --os-neutron-ml2-tenant-network-types=vxlan --os-neutron-ml2-type-drivers=vxlan,flat,vlan --os-neutron-l2-agent=openvswitch --os-neutron-ovs-bridge-mappings=extnet:br-ex --os-neutron-ovs-bridge-interfaces=br-ex:eth0 --nova-libvirt-virt-type=kvm</code> (replace {set password} and ''eth0'' with appropriate values)</li> | ||
+ | <li> Skip down to ''Now, create the external network with Neutron.'' and follow guidance.</li> | ||
+ | <li> Next move to ''Then create a router and set its gateway using the external network...'' and follow remaining guidance.</li> | ||
+ | </ul> | ||
+ | <li> Set up the floating IP addresses to the external network.</li> | ||
+ | <li> Create flavors named ''amahi.small'' of 10GB and ''amahi.medium'' of 15GB. | ||
+ | <li> Add the following rules to ''default'' Security Group (Ingress, IPv4, Remote IP 0.0.0.0): | ||
+ | <ul> | ||
+ | <li> ICMP / Any</li> | ||
+ | <li> TCP / 22</li> | ||
+ | <li> TCP / 80</li> | ||
+ | <li> TCP / 3000</li> | ||
+ | <li> TCP / 443 (<i>Optional</i>)</li> | ||
+ | </ul> | ||
<li> Add users and private keys for SSH login</li> | <li> Add users and private keys for SSH login</li> | ||
<li> Disable SSH password and root login</li> | <li> Disable SSH password and root login</li> | ||
− | <li> | + | <li> Add users to web UI access</li> |
− | <li> | + | <li> Ensure OS updates applied and reboot</li> |
− | <li> | + | <li> Create a single instance with floating IP</li> |
− | <li> | + | <li> Verify all is working as expected.</li> |
− | <ul> | + | </ul> |
− | < | + | |
− | + | ||
− | + | <b>NOTE:</b> A terminal script has been developed to automate the installation steps above for OpenStack after CentOS is installed. | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
=Naming Convention= | =Naming Convention= | ||
+ | The following is recommended for standardization: | ||
* Instances: '''os-function''' (i.e. ''f24-bot'', ''f24-repo'', ''f24-dev'', etc) | * Instances: '''os-function''' (i.e. ''f24-bot'', ''f24-repo'', ''f24-dev'', etc) | ||
* Images: '''os-type''' (i.e. ''f24-cd'', ''f24-dvd'', etc) | * Images: '''os-type''' (i.e. ''f24-cd'', ''f24-dvd'', etc) | ||
* Snapshot: '''os-function-ss#''' (i.e. ''f24-repo-ss1'', ''f24-bot-ss2'', etc) | * Snapshot: '''os-function-ss#''' (i.e. ''f24-repo-ss1'', ''f24-bot-ss2'', etc) | ||
− | * Volumes: '''instance-vol''' (i.e. ''f24-repo-vol'', ''mirrormgr-vol'', '' | + | * Volumes: '''instance-vol''' (i.e. ''f24-repo-vol'', ''mirrormgr-vol'', ''dlmaster-vol'', etc) |
=Build Images= | =Build Images= | ||
− | This will outline how to build OpenStack images using Proxmox VE. | + | This will outline how to build OpenStack images using [https://www.proxmox.com/en/ Proxmox VE] (alternate method in development). |
* Log into Proxmox VE web UI | * Log into Proxmox VE web UI | ||
− | * Create a VM or clone an existing one | + | * Create a VM or clone an existing one (<code><i>type</i>=qemu64</code>) |
** If creating a VM, install the OS | ** If creating a VM, install the OS | ||
** If using a clone, start the VM | ** If using a clone, start the VM | ||
* Open a console window for the VM | * Open a console window for the VM | ||
− | ** Log in and as root do the following | + | ** Log in and as <code>root</code> do the following |
*** <code>dd if=/dev/zero of=/mytempfile bs=1M</code> (zero out any unused space) | *** <code>dd if=/dev/zero of=/mytempfile bs=1M</code> (zero out any unused space) | ||
*** <code>rm -f /mytempfile</code> | *** <code>rm -f /mytempfile</code> | ||
** Shutdown VM | ** Shutdown VM | ||
** Log into Proxmox VE via SSH and execute the following from command line | ** Log into Proxmox VE via SSH and execute the following from command line | ||
− | *** Navigate to <code>/ | + | *** Navigate to <code>/dev/pve</code> |
− | + | *** <code>qemu-img convert -O qcow2 original_image new_image.qcow2</code> | |
− | *** <code>qemu-img convert -O qcow2 original_image | + | *** Copy <code>new_image.qcow2</code> to a safe location for uploading into OpenStack |
− | *** Copy | ||
− | |||
*** Delete the VM from Proxmox VE web UI | *** Delete the VM from Proxmox VE web UI | ||
− | * Use | + | * Use WinSCP or similar program to copy the <code>new_image.qcow2</code> to client machine |
* Upload into OpenStack via the web UI | * Upload into OpenStack via the web UI | ||
− | ''' | + | '''Reference:''' [https://gazelle.ihe.net/book/export/html/917 Transfer Proxmox VM] |
=Create Instance= | =Create Instance= | ||
− | This is a nice straightforward tutorial on [https:// | + | This is a nice straightforward tutorial on [https://www.tecmint.com/create-deploy-and-launch-virtual-machines-in-openstack/ how to deploy and launch an instance (VM)]. |
+ | |||
+ | Also refer to [https://docs.openstack.org/mitaka/user-guide/cli_nova_launch_instance_using_ISO_image.html launch instance from ISO] for an alternate method. | ||
=Backup= | =Backup= | ||
− | '''Last Backup completed:''' | + | '''Last Backup completed:''' |
− | + | * Instances - ''04 Jul 2021'' | |
+ | * Volumes - ''04 Jul 2021'' | ||
+ | |||
+ | |||
+ | '''NOTES:''' | ||
* Back up scripts have been created to synchronize instances, volumes, and snapshots to a secondary drive on demand. | * Back up scripts have been created to synchronize instances, volumes, and snapshots to a secondary drive on demand. | ||
− | * | + | * Minimal of monthly back ups be done in case of catastrophic failure. |
+ | * See [https://github.com/Nexenture/openstack-instances-backups OpenStack Instance Backups] to automate via terminal script (some modification is needed). | ||
=Tips= | =Tips= | ||
+ | ==Command Line== | ||
+ | [https://docs.openstack.org/python-openstackclient/ussuri/ OpenStack Command Line Guide] | ||
==Network Issues== | ==Network Issues== | ||
If the DNS server is changed or networking appears inoperable, check the following to ensure the DNS is correct: | If the DNS server is changed or networking appears inoperable, check the following to ensure the DNS is correct: | ||
* ''/etc/sysconfig/network-scripts/ifcfg-br-ex'' | * ''/etc/sysconfig/network-scripts/ifcfg-br-ex'' | ||
− | * ''/etc/resolve'' | + | * ''/etc/resolve.conf'' |
Next restart neutron network services: | Next restart neutron network services: | ||
− | + | <code> | |
− | service neutron-dhcp-agent restart<br /> | + | :service neutron-server restart<br /> |
− | service neutron-l3-agent restart<br /> | + | :service neutron-dhcp-agent restart<br /> |
− | service neutron-metadata-agent restart | + | :service neutron-l3-agent restart<br /> |
− | service neutron-openvswitch-agent restart</code | + | :service neutron-metadata-agent restart<br /> |
+ | :service neutron-openvswitch-agent restart | ||
+ | </code> | ||
==Volume Issues== | ==Volume Issues== | ||
When a volume becomes detached and/or shows in error, the state can be reset: | When a volume becomes detached and/or shows in error, the state can be reset: | ||
− | + | <code> | |
− | : | + | :source keystonerc_admin |
+ | :cinder reset-state ''volume_id'' | ||
+ | </code> | ||
or use web UI. | or use web UI. | ||
Also refer to [https://bugs.amahi.org/issues/2051 Amahi Bug #2051]. | Also refer to [https://bugs.amahi.org/issues/2051 Amahi Bug #2051]. | ||
+ | |||
+ | ==Instance Issues== | ||
+ | When an instance '''Status''' reflects an ''error'' state, reset it by: | ||
+ | <code> | ||
+ | :source keystonerc_admin | ||
+ | :nova reset-state --active INSTANCE_ID | ||
+ | </code> | ||
+ | |||
+ | '''NOTE:''' Should the '''Power State''' reflect ''no state'' or blank, it can be corrected by doing the above step, then starting the instance. | ||
==Update/Reboot/Shutdown Process== | ==Update/Reboot/Shutdown Process== | ||
+ | Follow guidelines below to prevent corruption: | ||
* Shutdown/Disconnect | * Shutdown/Disconnect | ||
− | ** Stop all | + | ** Stop all instances |
− | + | ** Verify all instances stopped | |
− | ** Verify all | + | ** Perform Update/Reboot/Shutdown |
− | ** Perform Update/Reboot | + | <br> |
− | * Once system | + | * Once system Started/Rebooted: |
− | |||
− | |||
** Start all needed instances | ** Start all needed instances | ||
** Verify all instances are operational | ** Verify all instances are operational | ||
Line 135: | Line 172: | ||
* As root user, execute <code>source keystonerc_admin</code> | * As root user, execute <code>source keystonerc_admin</code> | ||
* Use the following to reserve the IP address: | * Use the following to reserve the IP address: | ||
− | + | <code> | |
+ | :openstack port create --network private --fixed-ip subnet_id=private_subnet,ip_address=x.x.x.x static-ip-1 | ||
+ | </code> | ||
* Create the instance and boot via command-line vs the web UI: | * Create the instance and boot via command-line vs the web UI: | ||
− | + | <code> | |
+ | :openstack server create --image IMAGE_NAME --flavor amahi.small --nic port-id=NETWORK_ID_FROM_ABOVE_COMMAND INSTANCE_NAME | ||
+ | </code> | ||
:'''NOTE:''' If image name has spaces, enclose it in double quotes. | :'''NOTE:''' If image name has spaces, enclose it in double quotes. | ||
Ref: [https://ask.openstack.org/en/question/30690/add-multiple-specific-ips-to-instance/ Add Multiple Specific IPs to Instance] | Ref: [https://ask.openstack.org/en/question/30690/add-multiple-specific-ips-to-instance/ Add Multiple Specific IPs to Instance] | ||
− | == | + | ==Change Instance Flavor== |
− | See [https:// | + | As ''root'' user: |
+ | <code> | ||
+ | :source keystonerc_admin | ||
+ | </code> | ||
+ | See [https://docs.openstack.org/newton/user-guide/cli-change-the-size-of-your-server Change the size of your server] for command line guidance. | ||
− | == | + | ==Fedora Cloud Base Images== |
− | + | See [https://alt.fedoraproject.org/cloud/ Fedora Cloud Base Images] for guidance. | |
− | ---- | + | |
+ | == Migration == | ||
+ | === Images === | ||
+ | See [https://raymii.org/s/tutorials/Openstack_Glance_Image_Download.html OpenStack Glance Image Download Tutorial] for how to transfer images from one server to another. | ||
+ | |||
+ | === Snapshots === | ||
+ | See [https://community.cisco.com/t5/cloud-blogs/how-to-snapshot-an-openstack-nova-instance-boot-it-download/ba-p/3661228 Snapshot an OpenStack instance, boot it, and download] for how to transfer snapshots from one instance to another. | ||
+ | |||
+ | === Volumes === | ||
+ | See [http://blog.oddbit.com/2015/09/29/migrating-cinder-volumes-between-openstack-environments/ Migrating cinder volumes between OpenStack environments] for how to transfer volumes from one instance to another. | ||
+ | |||
+ | ==Fix Broken Web UI== | ||
+ | Add the following line to ''/etc/httpd/conf.d/15-horizon_vhost.conf'': | ||
+ | WSGIApplicationGroup %{GLOBAL} | ||
+ | Then restart the web server | ||
+ | |||
+ | Ref: [https://ask.openstack.org/en/question/91352/openstack-mitaka-can-not-access-dashboard/ OpenStack cannot access Dashboard] | ||
+ | |||
+ | =Known Issues= | ||
+ | Cannot use multiple SSH keys via Dashboard ([https://bugs.launchpad.net/nova/+bug/1499751 OpenStack Bug #1499751]) | ||
+ | <br> |
Latest revision as of 02:23, 26 May 2022
Work In Progress | |
---|---|
This article is currently undergoing major expansion or restructuring. You are welcome to assist by editing it as well. If this article has not been edited in several days, please remove this template. |
Last Update: May 22, 2022
Contents
Objective
This is a project to update and modernize the infrastructure that keeps the Amahi web sites and services running.
The idea is to provide easier and more sustainable management of the infrastructure to leave more time for the team to devote to moving the project forward.
NOTE: this project is not about running Amahi platform software on virtual servers, etc. For that there is a separate page on Virtualization.
Goals
We have multiple goals:
- Operate internal build machines in a reliable, efficient way, so that we have consistent and updated builds/releases
- Consistent and recent backups making things recoverable
- Execute some testing of Amahi apps more easily and efficiently
- Test new features in an isolated manner
Configuration
Dell Server (F1DH):
- Dual Xeon E5450 3.0 GHz Processors
- Gigabit Network Interface
- KVM Network Interface
- RAID Controller
Server #1 (CURRENT):
- OpenStack release Yoga
- Operating System: CentOS 8 Stream
- 32GB PC2-5300 RAM (8x4)
- Quick Swap Drive Bays
- 1 - 1TB
- 2 - 750GB (Backup)
- 3 - EMPTY
- 4 - EMPTY
Server #2 (BACKUP):
- OpenStack release Wallaby
- Operating System: CentOS 8 Stream
- 24GB PC2-5300 RAM (4x2/8x2)
- Quick Swap Drive Bays
- 1 - 850GB (LVM)
- 2 - EMPTY
- 3 - 120GB SSD (Backup)
- 4 - EMPTY
Setup
The following guidance is based on the latest release of OpenStack (Yoga):
- Download CentOS 8 Stream minmal image and install following Installing CentOS 8 using NetBoot ISO Image tutorial (use LVM, but remove /home partition and redistribute space to /).
- Set Static IP Address and configure Fully Qualified Domain Name, then reboot
- Follow step 0 - 3 of the RDO Packstack Quickstart for the OpenStack installation (ensure step 0 prerequisites are completed first).
- At step 4, follow the Neutron with existing network guidance:
- Use command:
packstack --allinone --provision-demo=n --cinder-volumes-size=100G --keystone-admin-passwd={set password} --os-neutron-ml2-mechanism-drivers=openvswitch --os-neutron-ml2-tenant-network-types=vxlan --os-neutron-ml2-type-drivers=vxlan,flat,vlan --os-neutron-l2-agent=openvswitch --os-neutron-ovs-bridge-mappings=extnet:br-ex --os-neutron-ovs-bridge-interfaces=br-ex:eth0 --nova-libvirt-virt-type=kvm
(replace {set password} and eth0 with appropriate values) - Skip down to Now, create the external network with Neutron. and follow guidance.
- Next move to Then create a router and set its gateway using the external network... and follow remaining guidance.
- Set up the floating IP addresses to the external network.
- Create flavors named amahi.small of 10GB and amahi.medium of 15GB.
- Add the following rules to default Security Group (Ingress, IPv4, Remote IP 0.0.0.0):
- ICMP / Any
- TCP / 22
- TCP / 80
- TCP / 3000
- TCP / 443 (Optional)
- Add users and private keys for SSH login
- Disable SSH password and root login
- Add users to web UI access
- Ensure OS updates applied and reboot
- Create a single instance with floating IP
- Verify all is working as expected.
NOTE: A terminal script has been developed to automate the installation steps above for OpenStack after CentOS is installed.
Naming Convention
The following is recommended for standardization:
- Instances: os-function (i.e. f24-bot, f24-repo, f24-dev, etc)
- Images: os-type (i.e. f24-cd, f24-dvd, etc)
- Snapshot: os-function-ss# (i.e. f24-repo-ss1, f24-bot-ss2, etc)
- Volumes: instance-vol (i.e. f24-repo-vol, mirrormgr-vol, dlmaster-vol, etc)
Build Images
This will outline how to build OpenStack images using Proxmox VE (alternate method in development).
- Log into Proxmox VE web UI
- Create a VM or clone an existing one (
type=qemu64
)- If creating a VM, install the OS
- If using a clone, start the VM
- Open a console window for the VM
- Log in and as
root
do the followingdd if=/dev/zero of=/mytempfile bs=1M
(zero out any unused space)rm -f /mytempfile
- Shutdown VM
- Log into Proxmox VE via SSH and execute the following from command line
- Navigate to
/dev/pve
qemu-img convert -O qcow2 original_image new_image.qcow2
- Copy
new_image.qcow2
to a safe location for uploading into OpenStack - Delete the VM from Proxmox VE web UI
- Navigate to
- Log in and as
- Use WinSCP or similar program to copy the
new_image.qcow2
to client machine - Upload into OpenStack via the web UI
Reference: Transfer Proxmox VM
Create Instance
This is a nice straightforward tutorial on how to deploy and launch an instance (VM).
Also refer to launch instance from ISO for an alternate method.
Backup
Last Backup completed:
- Instances - 04 Jul 2021
- Volumes - 04 Jul 2021
NOTES:
- Back up scripts have been created to synchronize instances, volumes, and snapshots to a secondary drive on demand.
- Minimal of monthly back ups be done in case of catastrophic failure.
- See OpenStack Instance Backups to automate via terminal script (some modification is needed).
Tips
Command Line
Network Issues
If the DNS server is changed or networking appears inoperable, check the following to ensure the DNS is correct:
- /etc/sysconfig/network-scripts/ifcfg-br-ex
- /etc/resolve.conf
Next restart neutron network services:
- service neutron-server restart
- service neutron-dhcp-agent restart
- service neutron-l3-agent restart
- service neutron-metadata-agent restart
- service neutron-openvswitch-agent restart
Volume Issues
When a volume becomes detached and/or shows in error, the state can be reset:
- source keystonerc_admin
- cinder reset-state volume_id
or use web UI.
Also refer to Amahi Bug #2051.
Instance Issues
When an instance Status reflects an error state, reset it by:
- source keystonerc_admin
- nova reset-state --active INSTANCE_ID
NOTE: Should the Power State reflect no state or blank, it can be corrected by doing the above step, then starting the instance.
Update/Reboot/Shutdown Process
Follow guidelines below to prevent corruption:
- Shutdown/Disconnect
- Stop all instances
- Verify all instances stopped
- Perform Update/Reboot/Shutdown
- Once system Started/Rebooted:
- Start all needed instances
- Verify all instances are operational
Create Static IP for Instance
Setting a static IP for instances will ensure the internal IP address remains the same through it's life cycle. The floating IP address can be easily added afterwards.
- As root user, execute
source keystonerc_admin
- Use the following to reserve the IP address:
- openstack port create --network private --fixed-ip subnet_id=private_subnet,ip_address=x.x.x.x static-ip-1
- Create the instance and boot via command-line vs the web UI:
- openstack server create --image IMAGE_NAME --flavor amahi.small --nic port-id=NETWORK_ID_FROM_ABOVE_COMMAND INSTANCE_NAME
- NOTE: If image name has spaces, enclose it in double quotes.
Ref: Add Multiple Specific IPs to Instance
Change Instance Flavor
As root user:
- source keystonerc_admin
See Change the size of your server for command line guidance.
Fedora Cloud Base Images
See Fedora Cloud Base Images for guidance.
Migration
Images
See OpenStack Glance Image Download Tutorial for how to transfer images from one server to another.
Snapshots
See Snapshot an OpenStack instance, boot it, and download for how to transfer snapshots from one instance to another.
Volumes
See Migrating cinder volumes between OpenStack environments for how to transfer volumes from one instance to another.
Fix Broken Web UI
Add the following line to /etc/httpd/conf.d/15-horizon_vhost.conf:
WSGIApplicationGroup %{GLOBAL}
Then restart the web server
Ref: OpenStack cannot access Dashboard
Known Issues
Cannot use multiple SSH keys via Dashboard (OpenStack Bug #1499751)