Difference between revisions of "Infrastructure Virtualization Project"
Line 150: | Line 150: | ||
See [https://www.rdoproject.org/Running_an_instance#Step_5:_Launch_the_instance Launch Fedora Cloud images] for guidance. | See [https://www.rdoproject.org/Running_an_instance#Step_5:_Launch_the_instance Launch Fedora Cloud images] for guidance. | ||
− | == Transfer Images == | + | == Transfer == |
+ | === Images === | ||
See [https://raymii.org/s/tutorials/Openstack_Glance_Image_Download.html OpenStack Glance Image Download Tutorial] for how to transfer images from on server to another. | See [https://raymii.org/s/tutorials/Openstack_Glance_Image_Download.html OpenStack Glance Image Download Tutorial] for how to transfer images from on server to another. | ||
Revision as of 00:57, 13 October 2017
![]() |
Work In Progress |
---|---|
This article is currently undergoing major expansion or restructuring. You are welcome to assist by editing it as well. If this article has not been edited in several days, please remove this template. |
Contents
Objective
This is a project to update and modernize the infrastructure that keeps the Amahi web sites and services running.
The idea is to provide easier and more sustainable management of the infrastructure to leave more time for the team to devote to moving the project forward.
NOTE: this project is not about running Amahi platform software on virtual servers, etc. For that there is a separate page on Virtualization.
Goals
We have multiple goals:
- run some of internal build machines in a reliable, efficient way, so that we have consistent and updated builds/releases
- have consistent and recent backups making things recoverable
- run some testing of Amahi apps more easily and efficiently
- test new features in an isolated manner
Known Issues
- Cannot use multiple SSH keys via Dashboard (OpenStack Bug #1499751)
Hardware
Dell Rack Server
- Dual Xeon E5450 3.0 GHz Processors
- 32GB PC2-5300 RAM (8x4)
- Two Gigabit Network Interfaces
- KVM Network Interface
- RAID Controller
- Four Quick Swap Drive Bays
- 1 - 1 TB (OS)
- 2 - 120 GB SSD (Instances)
- 3 - 750 GB (Backup)
- 4 - 500 GB (Spare)
Software
- CentOS 7.2 x86_64 (Minimal)
- OpenStack Ocata Release
Setup
- Download CentOS 7.2 x86_64 minimal image and install following CentOS 7.2 Netinstall Guide tutorial.
- Configure FQDN (
/etc/hosts
and/etc/hostname
) - Add users and private keys for SSH login
- Disable SSH password and root login
- Follow step 0 - 3 of the RDO Packstack Quickstart for the OpenStack installation (ensure step 0 prerequisites are completed first).
- At the
packstack --allinone
step, follow the Neutron with existing network guidance instead (omit–os-neutron-ovs-bridge-interfaces=br-ex:eth0
argument). Networks can also be setup with the web UI (preferred). - Refer to floating IP range for setting up the floating IP addresses to the external network. This can also now be done with the web UI (preferred).
- Create flavor named amahi.small of 10GB.
- Extend cinder-volumes past 20GB:
- Followed the OpenStack increase volume capacity and Maintaining loopback devices after reboot tutorials (i.e. highlighted text)
- Results in 50Gb additional space for volumes.
- Total volume space available is now 70GB.
Naming Convention
- Instances: os-function (i.e. f24-bot, f24-repo, f24-dev, etc)
- Images: os-type (i.e. f24-cd, f24-dvd, etc)
- Snapshot: os-function-ss# (i.e. f24-repo-ss1, f24-bot-ss2, etc)
- Volumes: instance-vol (i.e. f24-repo-vol, mirrormgr-vol, dl-master-vol, etc)
Build Images
This will outline how to build OpenStack images using Proxmox VE.
- Log into Proxmox VE web UI
- Create a VM or clone an existing one
- If creating a VM, install the OS
- If using a clone, start the VM
- Open a console window for the VM
- Log in and as root do the following
dd if=/dev/zero of=/mytempfile bs=1M
(zero out any unused space)rm -f /mytempfile
- Shutdown VM
- Log into Proxmox VE via SSH and execute the following from command line
- Navigate to
/var/lib/vz/images/###
(number of VM) mv original_image.qcow2 original_image.qcow2_backup
(rename original image)qemu-img convert -O qcow2 original_image.qcow2_backup original_image.qcow2
- Copy new
.qcow2
image to a safe location for uploading into OpenStack - Remove
.backup
file - Delete the VM from Proxmox VE web UI
- Navigate to
- Log in and as root do the following
- Use WINScp or similar program to copy the
.qcow2
image to client machine - Upload into OpenStack via the web UI
REF: Reclaim disk space from .qcow2 or .vmdk image
Create Instance
This is a nice straightforward tutorial on Creating an instance.
Backup
- Last Backup completed:
- Instances - 31 Aug 2017
- Volumes - 4 Aug 2017
- Back up scripts have been created to synchronize instances, volumes, and snapshots to a secondary drive on demand.
- Minimal of monthly back ups be done in case of catastrophic failure.
- See OpenStack Instance Backups to automate via terminal script (some modification is needed).
Tips
Network Issues
If the DNS server is changed or networking appears inoperable, check the following to ensure the DNS is correct:
- /etc/sysconfig/network-scripts/ifcfg-br-ex
- /etc/resolve.conf
Next restart neutron network services:
service neutron-server restart
service neutron-dhcp-agent restart
service neutron-l3-agent restart
service neutron-metadata-agent restart
service neutron-openvswitch-agent restart
Volume Issues
When a volume becomes detached and/or shows in error, the state can be reset:
- source keystonerc_admin
- cinder reset-state volume_id
or use web UI.
Also refer to Amahi Bug #2051.
Instance Issues
When an instance Status reflects an error state, reset it by:
- source keystonerc_admin
- nova reset-state --active INSTANCE_ID
NOTE: Should the Power State reflect no state or blank, it can be corrected by doing the above step, then starting the instance.
Update/Reboot/Shutdown Process
- Shutdown/Disconnect
- Stop all instances
- Verify all instances stopped
- Perform Update/Reboot/Shutdown
- Once system Started/Rebooted:
- Start all needed instances
- Verify all instances are operational
Create Static IP for Instance
Setting a static IP for instances will ensure the internal IP address remains the same through it's life cycle. The floating IP address can be easily added afterwards.
- As root user, execute
source keystonerc_admin
- Use the following to reserve the IP address:
neutron port-create private_network --fixed-ip subnet_id=private_subnet,ip_address=x.x.x.x
- Create the instance and boot via command-line vs the web UI:
nova boot --image NAME_OF_IMAGE --flavor amahi.small --nic port-id=ID_FOR NETWORK_FROM_ABOVE_COMMAND_RESULTS VM_NAME
- NOTE: If image name has spaces, enclose it in double quotes.
Ref: Add Multiple Specific IPs to Instance
Fedora Cloud Images
See Launch Fedora Cloud images for guidance.
Transfer
Images
See OpenStack Glance Image Download Tutorial for how to transfer images from on server to another.
Transfer Snapshots
See Snapshot an OpenStack instance and download it for how to transfer snapshots from one instance to another.
Transfer Volumes
See Migrating cinder volumes between OpenStack environments for how to transfer volumes from one instance to another.
Fix Broken Web UI
Add the following line to /etc/httpd/conf.d/15-horizon_vhost.conf:
WSGIApplicationGroup %{GLOBAL}
Then restart the web server
Ref: OpenStack cannot access Dashboard
Miscellaneous
Refer to Amahi Bug #2050 for some OpenStack Command-Line syntax.