Changes

From Amahi Wiki
Jump to: navigation, search
6,227 bytes added ,  00:16, 7 September 2017
{{MessageBox|
backgroundcolor = #ffaaaa|
image =Warning.png|
heading =WARNING|
message = Development notes for Amahi 11 on Fedora 26.}}
Containers!
==Overview==
==Architecture Overview==
[[File:architecture.jpg]]
The idea of implementation is very similar to what is showing in the image above. Each app will run as a container. Each container will expose their own port// TODO - Add image hereports as can be seen attached to the apps above. We can map a host system port to the `exposed port` in the container. And then using reverse proxy we can connect different subdomains to different container apps.
The idea of implementation is very similar to what is showing For example in the image diagram above, ports 35001 to 35008 are ports on the host machine inside which containers (App1 to App4) run. Each app will run as a containerruns different services on different ports. Each container will expose their own port/ports as can For the apps to be seen as `exposed port` in accessible from systems outside the diagram above. We can map a host system port (`mapped port` in the diagram) we have to map the `exposed port` in ports on the container. And then using reverse proxy we can connect different subdomains to different container appswith those on the host.
This ===How it Works?===Lets assume we are trying to run multiple container apps. Let them be App1, App2, App3 and App4.  '''NOTE:''' Each app will have its own networking stack and hence can run a service on any port that they want.   App1 - 9000, 80 App2 - 8000, 80 App3 - 5000, 80 App4 - 22, 80  Each app has been explained a web server running on port 80 (let’s assume) and they have some other service running on different ports. There can be multiple services running on different ports. For example in detail the above figure each App has two services running on two ports.  We want the web interface of each of the app to be accessible to a user. Assume a user is running their HDA at IP address: 192.168.56.105 and domain amahi.net  They want to access App1 at app1.amahi.net, App2 at app2.amahi.net and so on. Inside the HDA each app is running inside an isolated environment with their own networking stack and hence they are not reachable from the outside world. So how do we work around this problem? The answer to that is reverse proxy. We have apache running on our HDA which handles all the requests coming on *.amahi.net. So for container apps we bind their web server port with a port on the host.   App1 -> 80 binds to -> 35002 on host App2 -> 80 binds to -> 35004 on host App3 -> 80 binds to -> 35006 on host App4 -> 80 binds to -> 35008 on host This “binding” is similar to port forwarding. Docker by default runs these containers on a separate network and each of those containers have an IP address (local, accessible only inside the host machine) and run their own networking stack.  Once the port binding is complete if we access -> ''<nowiki>http://192.168.56.105:35002</nowiki>'' we should get the web interface of App1 and so on. But for our users we want the app to be reachable at app1.amahi.net. So earlier the hda used to create a virtual host file for each app which used to run the relevant app server based on the url. For container apps, the present approach does this:  The apache running on host reverse proxies all requests coming on ''<nowiki>http://app1.amahi.net</nowiki>'' to ''<nowiki>http://localhost:35002</nowiki>'' , This way for the user the url is ''<nowiki>http://app1.amahi.net</nowiki>'' and they can still access the web interface of the app. Note that apache is running on port 80 in the Design Documenthost.
==How the app installation works?==
 
This section presents an overview of how installation works.
Assuming we have official container for an app available we can easily integrate them to amahi. If the official image is not available then we might have to build one of our own like I did for osticket and coppermine.
===Building Images===
Building images can be tricky and does require some knowledge of the apps as well (For example which php libraries to install, etc). There's a well defined procedure for building images for node and rails apps as well.
* [Dockerizing Node Apps](https://nodejs.org/en/docs/guides/nodejs-docker-webapp/)Dockerizing Node Apps] * [Dockerizing Rails Apps](https://semaphoreci.com/community/tutorials/dockerizing-a-ruby-on-rails-application)Dockerizing Rails Apps] 
Building images can be tricky and the image size is a very major issue. To reduce the image size I would suggest the readers to look up the following articles:
* [Reduce Docker Image Size](https://blog.codeship.com/reduce-docker-image-size/)Reduce Docker Image Size] * [Alpine Docker](https://hub.docker.com/_/alpine/)Alpine Docker]
===Install Script===
Once the image is available we can write an install script. You can see below a sample install script for gitlab.
```sh cat > docker-compose.yml << 'EOF' gitlab-container: image: 'gitlab/gitlab-ce:latest' container_name: "APP_IDENTIFIER" restart: unless-stopped hostname: 'APP_HOSTNAME' environment: GITLAB_OMNIBUS_CONFIG: | external_url <nowiki>'http://APP_HOSTNAME:HOST_PORT'</nowiki> gitlab_rails['gitlab_shell_ssh_port'] = 2224 ports: - 'HOST_PORT:HOST_PORT' - '2224:22' volumes: - './srv/gitlab/config:/etc/gitlab' - './srv/gitlab/logs:/var/log/gitlab' - './srv/gitlab/data:/var/opt/gitlab' EOF docker-compose up -d```
This script creates a docker-compose.yml file and then runs `docker-compose up -d` command which essentially creates and runs the container.
The `yml` file can have different parameters. For that we might have to refer to docker and docker-compose documentation. `restart : unless-stopped` is used to handle failovers of containers. If a container crashes for some reaosn then it will restart automatically.
===Understanding the script===
Container script for each app will have these parameters for sure:
```
container-name: "APP_IDENTIFIER"
restart : unless-stopped
ports:
- 'HOST_PORT:xyz' # Not required for apps which are not webapp
container-name: "APP_IDENTIFIER" restart : unless-stopped ports: - 'HOST_PORT:xyz' # Not required for apps which are not webapp # xyz = any port inside the container # HOST_PORT is port on the host machine. Two different containers cannot have same value for # HOST_PORT but can have the same value for xyz (Just to clear up the confusion about # HOST_PORT)```
Most of the configuration related to the container will be written in the docker-compose.yml file but some data we have to extract during run time like the HOST_PORT using which will reverse proxy to the container. HOST_PORT is 35000+app_id and has to be derived during app installation. Similarly APP_IDENTIFIER and APP_HOSTNAME are derived during runtime. The code below is used to put that data into install script during runtime.
```rb #app/models/app.rb Line 322 to 326 install_script = installer.install_script install_script = install_script.gsub(/HOST_PORT/, (BASE_PORT+self.id).to_s) install_script = install_script.gsub(/WEBAPP_PATH/, webapp_path) install_script = install_script.gsub(/APP_IDENTIFIER/, identifier) install_script = install_script.gsub(/APP_HOSTNAME/, app_host)```
Sample uninstall script
```sh
docker-compose stop
docker-compose rm -f
# Not removing the image. Just stopping the container.
```
docker-compose stop docker-compose rm -f # Not removing the image. Just stopping the container.  For most containers the above uninstallation script will work fine. This stops the running container and removes it. Please note that this doesn't delete any of the volumes attached (persistent storage. Please refer to docker documentatio documentation for more details regarding volumes) with the container so if you add a volume during installation (as we have done in the gitlab example above) then we have to remove them here during uninstallation. For example if we were to remove gitlab completely along with all files that were added by gitlab container then the uninstall script would look something like this:
```sh docker-compose stop docker-compose rm -f rm -rf srv # Removing the srv folder which holds the persistent files for gitlab container```
This behaviour behavior might not be intended for all applications. Right now I haven't removed static files for any apps that I have added.
===Reverse Proxy===
====Why is it needed?====
Each app will be running insidea container and they will attach Please refer to some port/ports on the host machine. For example, imagine if you are running an app like gitlab in container. Gitlab might require different ports for different things. Inside the container different services can run on any port as containers run in an isolated environment and have their own network stack. But ultimately the services running on these containers have [[#Architecture Overview]] to be reachable from the outside world. To make that possible we have to bind a port inside the container with a host port.  Gitlab App - Inside Container - Web Server Running on port 80We have to bind this with a port on host machine (our HDA). Let's assume we bind understand why it to port 35001. Now typing amahi.net:35001/ in the browser will open gitlab website. OSticket app - Inside Container - Web Server Running on port 9000Let's bind this to port 35002 on host machine. Now typing amahi.net:35002/ in the browser will open osticket website.  As we can see that each of these apps web interface is accessible on different ports but we want to achieve a behaviour like:  osticket.amahi.net -> opens osticket which is running on amahi.net:35002/gitlab.amahi.net -> opens gitlab which is running on amahi.net:35001/ But the url end point should be the same, i.e osticket.amahi.net and gitlab.amahi.net . To attain this behaviour we have to use Reverse Proxyneeded.  
For reverse proxy I have added a new app-container.conf file which can be seen below. The `APP_PORT` is changed during runtime.
```sh <VirtualHost *:80> ServerName HDA_APP_NAME ServerAlias HDA_APP_NAME.HDA_DOMAIN APP_ALIASES APP_CUSTOM_OPTIONS ProxyPreserveHost On ProxyPass / <nowiki>http://localhost:APP_PORT/</nowiki> ProxyPassReverse / <nowiki>http://localhost:APP_PORT/</nowiki> ErrorLog APP_ROOT_DIR/logs/error_log CustomLog APP_ROOT_DIR/logs/access_log combined env=!dontlog </VirtualHost>
ProxyPass / http://localhost:APP_PORT/
ProxyPassReverse / http://localhost:APP_PORT/
 
ErrorLog APP_ROOT_DIR/logs/error_log
CustomLog APP_ROOT_DIR/logs/access_log combined env=!dontlog
</VirtualHost>
 
```
APP_PORT part is derived from the app id. After installation the app will have some id in the database base. The APP_PORT will be 35000+app_id
Please note that we can run any kind of app in container. It might be a headless app and it might be a webapp. In case of web applications we need to define an external port (mapped port on host - Refer to Architecture Overview section) through which the app will be bind. Then to reach that app we have to reverse proxy. "APP_PORT" is essentially that. For apps which don't require a web interface we might not use this file at all.
'''NOTE''': HOST_PORT and APP_PORT are basically same thing but in the code they are used at two different places with different names to avoid confusion as per the context. For example in the Reverse Proxy case, APP_PORT variable is present inside the file `app-container.conf`. Here we want to reverse proxy to an app running on some port so it was named as APP_PORT.For the HOST_PORT , it was used in app/models/app.rb file, their it represents the port to be used on the host machine and hence named that way. But for one particular app, value of HOST_PORT=APP_PORT.
## ==How to add a new app? ==Taking example of [Hydra](https://hub.docker.com/r/linuxserver/hydra/)Hydra]
See the usage as mentioned by the maintainer:
```sh docker create --name=hydra \ -v <path to data>:/config \ -v <nzb download>:/downloads \ -e PGID=<gid> -e PUID=<uid> \ -e TZ=<timezone> \ -p 5075:5075 linuxserver/hydra```
Convert the above to a docker-compose file. Ignore the `-e PGID=<gid> -e PUID=<uid>`, even though it's relevant, it is out of the scope of this discussion.
``` hydra-container: image: 'docker.io/linuxserver/hydra' container_name: "hydra" restart: unless-stopped ports: - '5075:5075' volumes: - './config:/config' - './downloads:/downloads' - '/etc/localtime:/etc/localtime:ro' # Understanding the volume mounts: # ./config:/config -> As seen in the docker create commnad the -v command mentions the volumes. # Path to data that we are providing is a relative path. Every installed app has a path in which the # install script runs. So "config" and "downloads" folder will be created there in that path. # /etc/localtime:/etc/localtime:ro -> This is to make sure that the container uses the same time as used # by the host system. To avoid this mount we can also use # environment: # - TZ=<timezone> # in the docker compose file
# Understanding '''NOTE:''' Please note that adding apps might require knowledge about docker and docker-compose and discussing those is out of the scope of this documentation though the volume mountslinks mentioned below might be useful.* [https:# //docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/config#container-and-layers Container and Layers]* [https:/config -> As seen in the /docs.docker create commnad the -v command mentions the volumes. com/compose/overview/ Docker Compose]# Path to data that we are providing is a relative path* [https://docs. Every installed app has a path in which the# install script runsdocker. So "config" and "downloads" folder will be created there in that pathcom/engine/admin/start-containers-automatically/ Restart Policies].
# /etc/localtime:/etc/localtime:ro -> This is to make sure that the container uses the same time as used
# by the host system. To avoid this mount we can also use
# environment:
# - TZ=<timezone>
# in the docker compose file
```
NOTE: Please note that adding apps might require knowledge about docker and docker-compose and discussing those is out of the scope of this documentation though the links mentioned below might be useful. * [Container and Layers](https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/#container-and-layers) * [Docker Compose](https://docs.docker.com/compose/overview/) * [Restart Policies](https://docs.docker.com/engine/admin/start-containers-automatically/). Apps run as containers which are managed by docker. If docker daemon is shut down or stopped then the app will also stop. If the container crashes for some reason then it has to be restarted. Using restart policies we can manage this.
Now once we are done with making a docker-compose file we can test it on our local system to see if it is working properly or not. Once that's done, we can go ahead and add this image to amahi.org
For adding to amahi.org some modifications have to be done. The final changes can be seen below. Notice the `APP_IDENTIFIER` and `HOST_PORT` (For more info on this refer to "Understandin the script" section)
``` hydra-container: image: 'docker.io/linuxserver/hydra' container_name: "APP_IDENTIFIER" restart: unless-stopped ports: - 'HOST_PORT:5075' volumes: - './config:/config' - './downloads:/downloads' - '/etc/localtime:/etc/localtime:ro'```
The final install and uninstall scripts to be added on amahi.org will be
Install Script
```sh cat > docker-compose.yml << 'EOF' hydra-container: image: 'docker.io/linuxserver/hydra' container_name: "APP_IDENTIFIER" restart: unless-stopped ports: - 'HOST_PORT:5075' volumes: - './config:/config' - './downloads:/downloads' - '/etc/localtime:/etc/localtime:ro' EOF docker-compose up -d```
Uninstall Script
```sh docker-compose stop docker-comose rm -f rm -rf config # Use this if you want all files to be removed after uninstall rm -rf downloads # Use this if you want all files to be removed after uninstall```## NOTEFor more examples of install scripts and docker-compose files of different apps please refer to [https://github.com/vik-y/amahi_images amahi_images repo.] or the table given below.
You should be careful about the following{| class="wikitable"|+ <big>'''App examples'''</big>! Example app ! install-script ! Image repository|-| coppermine||https: //github.com/vik-y/amahi_images/tree/master/coppermine||https://github.com/vik-y/amahi_images/tree/master/coppermine|-| fengoffice||https://github.com/vik-y/amahi_images/tree/master/fengoffice||https://github.com/vik-y/amahi_images/tree/master/fengoffice|-| osticket||https://github.com/vik-y/amahi_images/tree/master/osticket||https://github.com/vik-y/amahi_images/tree/master/osticket|-| gitlab||https://github.com/vik-y/amahi_images/tree/master/gitlab||https://hub.docker.com/r/gitlab/gitlab-ce|-| radarr||https://github.com/vik-y/amahi_images/tree/master/radarr||https://github.com/linuxserver/docker-radarr|-| hydra||https://github.com/vik-y/amahi_images/tree/master/hydra||https://github.com/linuxserver/docker-hydra|-| logitech-media-server||https://github.com/vik-y/amahi_images/tree/master/logitech-media-server||https://github.com/larsks/docker-image-logitech-media-server|-| markdown-editor||https://github.com/vik-y/amahi_images/tree/master/markdown-editor||https://hub.docker.com/r/v4tech/markdown-editor|-| nzbget||https://github.com/vik-y/amahi_images/tree/master/nzbget||https://github.com/linuxserver/docker-nzbget|-| portainer||https://github.com/vik-y/amahi_images/tree/master/portainer||https://hub.docker.com/r/portainer/portainer/|-| redmine||https://github.com/vik-y/amahi_images/tree/master/redmine||https://hub.docker.com/_/redmine/|-| sinusbot||https://github.com/vik-y/amahi_images/tree/master/sinusbot||https://github.com/vik-y/amahi_images/tree/master/sinusbot|-| sonarr||https://github.com/vik-y/amahi_images/tree/master/sonarr||https://github.com/linuxserver/docker-sonarr|-| squid||https://github.com/vik-y/amahi_images/tree/master/squid||https://hub.docker.com/r/sameersbn/squid|}
## Future Work==Brief Overview of Code==The installation process remains the same and follows the same flow of control as before.
script/install-ap calls install_bg function in app.rb and that takes care of the app installation. In `install_bg` function we check if the app type is container and if yes then the rest is managed by `install_container_app` function. If it's not a container type app then see the else statement, where we just create a `webapp` for normal app.  if installer.kind.include? "container" #Try installing the container app #Mark the installation as failed if it returns false # For container app webapp creation will be handled inside the "install_container_app" function if !install_container_app(installer, webapp_path, identifier) self.install_status = 999 end else # Create the webapp normally if its not a container app self.create_webapp(:name => name, :path => webapp_path, :deletable => false, :custom_options => installer.webapp_custom_options, :kind => installer.kind) endFor container type apps the `webapp` creation is handled by the `install_container_app` function itself. Please note the way we are checking if an app is of container type? Notice : `if installer.kind.include? "container"`. The idea is that we might need to handle installation of different kind of container apps differently. So on amahi.org while adding container apps, for the kind field we have to use something like : `container-php5`, `container-python`, `container-custom`, etc. So to check if an app is container type we have to see if the `installer.kind` includes "container" pattern inside it.  ==Future Work=====Cleanup - too many images. ===
Presently when a containerised app is uninstalled the container is stopped and removed but the image used to run the container stays (Note the difference between container and image). I am not removing the image since images are generally large and what if users decides to reinstall the app? This has to be thought of more. Adding the feature of deleting image after installation is essentially 1 line of code. Going forward we might have to come up with a mechanism for clearing images which haven't been used for a long time.
### ===Reducing Download Size===
We can't build and maintain containers for all apps. Vendors provide official containers but some of them are huge in size. To fix this problem we can reduce the download to that of a normal installation by using an apt-cache, gem server or npm server on hda itself.
The idea is following: Instead of we building images on our server we can push the Dockerfile to the client and the client can build the docker image. While building the image they will download the required packages, gem files, node dependencies or whatever. If we have a mechanism to cache this download so that all the subsequent builds can use this data then we can save a lot of Internet usage. One possible way of doing this was running apt-cache, gem server and npm server on the client itself.
### ===Updating apps=== With containers updates can be really easy. We can support single click update of apps. It's just One possible way of implementing this feature would be to create an update button which on click would stop the running container, download the latest image of the app (which is the updated app) and restart the container. Though implementing this from docker perspective is not a matter big deal at all but it can get tricky as well if there are some major upgrades in code then the vendor must take care of supporting them nowdata migration from older version to newer versions in their image itself. But this problem of migration exists without containers as well.
### ===Support for advance configuration.===
With containers we can limit the cpu/memory/disk usage of each app. We just have to modify the docker-compose.yml file for the app for these changes without doing any modification to the source code. If required, this can be used.
### ===Collecting Logs===
It's a new feature, we will need to collect a lot of metrics from the users to understand how this feature is working and how it can be improved. Some of those features include
* The CPU info of systems which are running amahiAmahi. * RAM and Storage information. * Logs of containerised containerized apps to debug errors.
12,424

edits