Infrastructure Virtualization Project

From Amahi Wiki
Jump to: navigation, search

Objective

This is a project to update and modernize the infrastructure that keeps the Amahi web sites and services running.

The idea is to provide easier and more sustainable management of the infrastructure to leave more time for the team to devote to moving the project forward.

NOTE: this project is not about running Amahi platform software on virtual servers, etc. For that there is a separate page on Virtualization.

Goals

We have multiple goals:

  • Operate internal build machines in a reliable, efficient way, so that we have consistent and updated builds/releases
  • Consistent and recent backups making things recoverable
  • Execute some testing of Amahi apps more easily and efficiently
  • Test new features in an isolated manner

Configuration

Dell Server (F1DH):

  • Dual Xeon E5450 3.0 GHz Processors
  • Gigabit Network Interface
  • KVM Network Interface
  • RAID Controller


Operating System (OS): CentOS 7.6.1810

Server #1 (CURRENT):

  • OpenStack release Stein
  • 32GB PC2-5300 RAM (8x4)
  • Quick Swap Drive Bays
    • 1 - 1 TB
    • 2 - 120 GB SSD
    • 3 - 750 GB (Backup)
    • 4 - EMPTY


Server #2 (BACKUP):

  • OpenStack release Rocky
  • 24GB PC2-5300 RAM (4x2/8x2)
  • Quick Swap Drive Bays
    • 1 - 128GB SSD (LVM)
    • 2 - 800GB HD (LVM)
    • 3 - EMPTY
    • 4 - EMPTY


Setup

  • Download CentOS minmal image and install following CentOS 7.2 Netinstall Guide tutorial.
  • Configure Fully Qualified Domain Name and reboot
  • Follow step 0 - 2 of the RDO Packstack Quickstart for the OpenStack installation (ensure step 0 prerequisites are completed first).
  • At step 3, follow the Neutron with existing network guidance:
    • Use command packstack --allinone --provision-demo=n --os-neutron-ovs-bridge-mappings=extnet:br-ex --os-neutron-ovs-bridge-interfaces=br-ex:eth0 --os-neutron-ml2-type-drivers=vxlan,flat (replace eth0 with appropriate interface)
    • Skip down to Now, create the external network with Neutron. and follow guidance.
    • Next move to Then create a router and set its gateway using the external network... and follow remaining guidance.
  • Set up the floating IP addresses to the external network using web UI.
  • Create flavors named amahi.small of 10GB and amahi.medium of 15GB.
  • Add the following rules to default Security Group (Ingress, IPv4, Remote IP 0.0.0.0):
    • ICMP / Any
    • TCP / 22
    • TCP / 80
    • TCP / 3000
  • Extend cinder-volumes past 20GB using disk drive partition
  • Add users and private keys for SSH login
  • Disable SSH password and root login
  • Add users to web UI access
  • Install openstack-utils
  • Ensure OS updates applied and reboot
  • Create a single instance with floating IP
  • Verify all is working as expected.

Naming Convention

  • Instances: os-function (i.e. f24-bot, f24-repo, f24-dev, etc)
  • Images: os-type (i.e. f24-cd, f24-dvd, etc)
  • Snapshot: os-function-ss# (i.e. f24-repo-ss1, f24-bot-ss2, etc)
  • Volumes: instance-vol (i.e. f24-repo-vol, mirrormgr-vol, dl-master-vol, etc)

Build Images

This will outline how to build OpenStack images using Proxmox VE.

  • Log into Proxmox VE web UI
  • Create a VM or clone an existing one
    • If creating a VM, install the OS
    • If using a clone, start the VM
  • Open a console window for the VM
    • Log in and as root do the following
      • dd if=/dev/zero of=/mytempfile bs=1M (zero out any unused space)
      • rm -f /mytempfile
    • Shutdown VM
    • Log into Proxmox VE via SSH and execute the following from command line
      • Navigate to /var/lib/vz/images/### (number of VM)
      • mv original_image.qcow2 original_image.qcow2_backup (rename original image)
      • qemu-img convert -O qcow2 original_image.qcow2_backup original_image.qcow2
      • Copy new .qcow2 image to a safe location for uploading into OpenStack
      • Remove .backup file
      • Delete the VM from Proxmox VE web UI
  • Use WINScp or similar program to copy the .qcow2 image to client machine
  • Upload into OpenStack via the web UI


Reference: Reclaim disk space from .qcow2 or .vmdk image

Create Instance

Backup

  • Last Backup completed:
    • Instances - 28 Dec 2018
    • Volumes - 28 Dec 2018
  • Back up scripts have been created to synchronize instances, volumes, and snapshots to a secondary drive on demand.
  • Minimal of monthly back ups be done in case of catastrophic failure.
  • See OpenStack Instance Backups to automate via terminal script (some modification is needed).

Tips

Network Issues

If the DNS server is changed or networking appears inoperable, check the following to ensure the DNS is correct:

  • /etc/sysconfig/network-scripts/ifcfg-br-ex
  • /etc/resolve.conf


Next restart neutron network services:

service neutron-server restart
service neutron-dhcp-agent restart
service neutron-l3-agent restart
service neutron-metadata-agent restart
service neutron-openvswitch-agent restart

Volume Issues

When a volume becomes detached and/or shows in error, the state can be reset:

source keystonerc_admin
cinder reset-state volume_id

or use web UI.

Also refer to Amahi Bug #2051.

Instance Issues

When an instance Status reflects an error state, reset it by:

source keystonerc_admin
nova reset-state --active INSTANCE_ID

NOTE: Should the Power State reflect no state or blank, it can be corrected by doing the above step, then starting the instance.

Update/Reboot/Shutdown Process

  • Shutdown/Disconnect
    • Stop all instances
    • Verify all instances stopped
    • Perform Update/Reboot/Shutdown
  • Once system Started/Rebooted:
    • Start all needed instances
    • Verify all instances are operational

Create Static IP for Instance

Setting a static IP for instances will ensure the internal IP address remains the same through it's life cycle. The floating IP address can be easily added afterwards.

  • As root user, execute source keystonerc_admin
  • Use the following to reserve the IP address:
neutron port-create private_network --fixed-ip subnet_id=private_subnet,ip_address=x.x.x.x
  • Create the instance and boot via command-line vs the web UI:
nova boot --image NAME_OF_IMAGE --flavor amahi.small --nic port-id=ID_FOR NETWORK_FROM_ABOVE_COMMAND_RESULTS VM_NAME
NOTE: If image name has spaces, enclose it in double quotes.

Ref: Add Multiple Specific IPs to Instance

Change Instance Flavor

As root user:

source keystonerc_admin

See Change the size of your server for command line guidance.

Fedora Cloud Images

See Launch Fedora 23 Cloud image for guidance.

Migration

Images

See OpenStack Glance Image Download Tutorial for how to transfer images from one server to another.

Snapshots

See Snapshot an OpenStack instance, boot it, and download for how to transfer snapshots from one instance to another.

Volumes

See Migrating cinder volumes between OpenStack environments for how to transfer volumes from one instance to another.

Fix Broken Web UI

Add the following line to /etc/httpd/conf.d/15-horizon_vhost.conf: 

WSGIApplicationGroup %{GLOBAL}

Then restart the web server

Ref: OpenStack cannot access Dashboard

Miscellaneous

Refer to Amahi Bug #2050 for some OpenStack Command-Line syntax.

Known Issues


Retrieved from "https://wiki.amahi.org/index.php?title=Infrastructure_Virtualization_Project&oldid=112641"