Difference between revisions of "Greyhole"

From Amahi Wiki
Jump to: navigation, search
 
(144 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
[[Category:apps]]
 
[[Category:apps]]
−
<div style="border: 1px solid #A3B1BF; padding: .8em 1em; background-color: #E6F2FF; margin: 0px 1em;">
+
== What is Greyhole ==
−
'''NEW:''' [https://github.com/gboudreau/Greyhole/wiki/MigrateFromWHS Widows Home Server Drive Extender Migration Guide]
 
  
−
'''NEW:''' [[Greyhole troubleshooting]] guide
+
[http://www.greyhole.net/ Greyhole] is Amahi's [[Storage pooling]] technology of choice. Storage Pooling combines the space of multiple disk drives and makes them look as if they were all part of a single pool of disk space. Greyhole was designed independently of Amahi, however, with feedback and great work, the Greyhole team made it suitable for integration with Amahi.
−
</div>
 
  
−
 
−
== What is Greyhole ==
 
−
 
−
[http://www.greyhole.net/ Greyhole] is Amahi's [[Storage pooling]] technology. Storage Pooling combines the space of multiple disk drives and makes them look as if they were all part of a single pool of disk space.
 
 
Specifically, Greyhole:
 
Specifically, Greyhole:
 
* Combines the space from multiple drives into a single volume
 
* Combines the space from multiple drives into a single volume
 
* Distributes files across all drives in the pool
 
* Distributes files across all drives in the pool
 
* Creates multiple copies of files in admin-specified shares
 
* Creates multiple copies of files in admin-specified shares
 +
<br />
 +
Recommend you consult [https://github.com/gboudreau/Greyhole/wiki/WhatNotToPutInGreyholeShares what not to put in Greyhole shares] as a precaution when determining which shares to select. 
  
−
This feature is installed by default, but requires a few simple steps to begin utilizing the features.
+
You can add additional drives to your HDA and prepare them for use in Greyhole by following [[Adding_a_second_hard_drive_to_your_HDA|adding a second hard drive to your HDA]] guidance or use the Amahi [https://www.amahi.org/apps/disk-wizard Disk Wizard] (Amahi 8 or greater).
  
−
You can add additional drives to your hda and prepare them for use in Greyhole by following [[Adding a second hard drive to your HDA|this tutorial]].
+
{{Warning|
 +
<div style<nowiki>=</nowiki>"color:darkred">You should <u>'''NEVER'''</u> change or delete files in the shared directories (''/var/hda/files/*'' by default) directly on the HDA, using the terminal, or the Gnome file browser, for the shares for which you checked the ''Uses pool'' option in the Amahi dashboard. If you want to work with files on Greyhole shares on the HDA, you should [[Mount Shares Locally]].<br/>
 +
<br/>
 +
You should be safe reading existing files directly, or adding new files directly, as long as don't care that your new files will only be moved into the storage pool during the next check, which runs automatically at midnight (or manually using ''greyhole --fsck''). Until then, your new files will be stored in your shared directories (normally ''/var/hda/files/share_name/'').
  
−
To add a drive to your Greyhole storage pool. Open your HDA dashboard and navigate to the Shares tab. Then click on the Storage Pool sub-category. You will see a page similar to the one below.
+
Also, touching anything inside the '''gh''' directories that Greyhole creates at the root of your partitions is a recipe for disaster. [[Greyhole not on root|We strongly discourage you from using the root partition in a drive pool]].</div><br/>}}
  
−
== Important Warnings ==
+
'''NOTE:''' All commands are executed as root user (Fedora) or preceded with sudo (Ubuntu).
−
<div style="color:darkred">You should '''never''' change or delete files in the shared directories (''/var/hda/files/*'' by default) directly on the HDA, using the terminal, or the Gnome file browser, for the shares for which you checked the ''Uses pool'' option in the Amahi dashboard. If you want to work with files on Greyhole shares on the HDA, you should [[Mount Shares Locally]].<br/>
 
−
<br/>
 
−
You should be safe reading existing files directly, or adding new files directly, as long as don't care that your new files will only be moved into the storage pool during the next check, which runs automatically at midnight (or manually using ''greyhole --fsck''). Until then, your new files will be stored in your shared directories (normally ''/var/hda/files/share_name/'').
 
  
−
Also, touching anything inside the '''gh''' directories that Greyhole creates at the root of your partitions is a recipe for disaster.  [[Greyhole not on root | We strongly discourage you from using the root partition in a drive pool]].
+
== Amahi 7 or greater ==
−
</div><br/>
+
Amahi application [https://www.amahi.org/apps/greyhole-ui Greyhole UI] provides this capability.  Refer to the [[Greyhole_UI_App_Transition_Guide|Greyhole UI Application Transition guide]] for details on installation and configuration.
  
−
== Forcing an fsck ==
+
'''NOTE:''' If you would still like to setup your Greyhole pool manually, please see [[Amahi_Create_Manual_Greyhole_Setup|Manual Greyhole Setup]].
−
{{code|greyhole -fsck
 
−
greyhole -f}}
 
−
will run an fsck ONLY if the greyhole or samba configurations have changed. <br />
 
−
{{code|greyhole --fsck}}
 
−
will force a fsck.
 
  
−
== First things first ==
+
== Amahi 6 ==
  
−
You need to go in the Settings tab, and enable '''Advanced Settings'''.<br/>
+
You need to go in the Settings tab, and select [[Advanced_Settings|Enable Advanced Settings]]. Without Advanced Settings enabled, you won't see the following page and options.
−
Without Advanced Settings enabled, you won't see the following page and options.
 
  
−
== For Storage Pool (Partitions) ==
+
<b><u>Storage Pool</u></b><br />
  
 
[[File:Storage_Pool_1.png|600px]]
 
[[File:Storage_Pool_1.png|600px]]
Line 47: Line 37:
 
Select the drives you want available for your storage pool.
 
Select the drives you want available for your storage pool.
  
−
== Greyhole options ==
+
<b><u>Share Options</u></b><br />
  
 
The next step is to select the share you want to replicate across the pool.
 
The next step is to select the share you want to replicate across the pool.
Line 57: Line 47:
 
Check the option for "Uses pool" and choose the number of drives to replicate this share. In this instance we have chosen to use all drives available to the greyhole pool.
 
Check the option for "Uses pool" and choose the number of drives to replicate this share. In this instance we have chosen to use all drives available to the greyhole pool.
  
−
Note: Greyhole is not a backup solution. If a file is removed, all copies are removed.
+
'''NOTE:''' Greyhole is not a backup solution. If a file is removed, all copies are removed.
 +
 
 +
== Advanced Options ==
 +
 
 +
You can further configure Greyhole by manually editing '''/var/hda/platform/html/config/greyhole.yml'''. 
  
−
== Greyhole advanced options ==
+
:'''NOTE:''' Options duplicated in ''/etc/greyhole.conf will be overridden by grehole.yml.
  
−
You can further configure Greyhole by manually editing the following file:<br/>
+
To see what options are available, and what they do, refer to the [https://github.com/gboudreau/Greyhole/blob/master/greyhole.example.conf sample greyhole.conf] provided with Greyhole.
−
'''/var/hda/platform/html/config/greyhole.yml'''
 
  
−
To see what options are available, and what they do, refer to the [https://gist.github.com/1379102 sample greyhole.conf] provided with Greyhole.
+
greyhole.yml is in in [http://en.wikipedia.org/wiki/YAML YAML] format. Most of it should be easy enough to modify. The only exception would be the sticky_files (and optional sticky_into) options.<br/>
  
−
greyhole.yml is in in YAML format. Most of it should be easy enough to modify. The only exception would be the sticky_files (and optional sticky_into) options.<br/>
 
 
Here's an example of how those should appear:
 
Here's an example of how those should appear:
  
 
To specify the following to Greyhole in greyhole.conf:
 
To specify the following to Greyhole in greyhole.conf:
  
−
{{Text|Text=sticky_files <nowiki>=</nowiki> Music/
+
<pre>sticky_files <nowiki>=</nowiki> Music/
 
  sticky_files <nowiki>=</nowiki> Videos/Movies/
 
  sticky_files <nowiki>=</nowiki> Videos/Movies/
 
     stick_into <nowiki>=</nowiki> /mnt/hdd1/gh
 
     stick_into <nowiki>=</nowiki> /mnt/hdd1/gh
 
     stick_into <nowiki>=</nowiki> /mnt/hdd5/gh
 
     stick_into <nowiki>=</nowiki> /mnt/hdd5/gh
 
  sticky_files <nowiki>=</nowiki> Backups/CrashPlan/
 
  sticky_files <nowiki>=</nowiki> Backups/CrashPlan/
−
     stick_into <nowiki>=</nowiki> /mnt/hdd0/gh}}
+
     stick_into <nowiki>=</nowiki> /mnt/hdd0/gh</pre>
  
 
one should specify this in the greyhole.yml file, where '''precise indentation matters''':
 
one should specify this in the greyhole.yml file, where '''precise indentation matters''':
  
−
{{Text|Text=sticky_files:  
+
<pre>sticky_files:  
 
  - - Music/
 
  - - Music/
 
  - - Videos/Movies/
 
  - - Videos/Movies/
Line 86: Line 78:
 
     - /mnt/hdd5/gh
 
     - /mnt/hdd5/gh
 
  - - Backups/CrashPlan/
 
  - - Backups/CrashPlan/
−
   - - /mnt/hdd0/gh}}
+
   - - /mnt/hdd0/gh</pre>
−
 
 
−
For your changes to be effective, you'll need to force Amahi to regenerate the greyhole.conf file. You can do so by editing a share, clicking it's path, and just clicking the Save button without actually changing the path.
 
−
 
 
−
== Copying data into shares for the first time ==
 
−
 
 
−
When you start using Greyhole, you might want to copy or move all your existing files into your new shares that use the storage pool.
 
−
 
 
−
Note: This is not necessary if your files are already in Amahi shares. If they are there, and you enable the ''Uses pool'' option in the Amahi dashboard, the files will start getting moved around into the drives in your storage pool during the night (starting at midnight), when the nightly storage pool check starts.<br/>
 
−
The instructions below are for users who have more data to copy into the Greyhole shares than their currently free space in the /var/hda/files/ folder.
 
−
 
 
−
One way to do that is to mount the shares that use the Greyhole storage pool, either on the HDA itself, or on a client computer on your local network, and copy your data from their existing location into the mounted shares.
 
−
This can be time consuming, but it is the safest way to use Greyhole.
 
−
 
 
−
Another way to move your files from their current location into the storage pool is to share their current location using an Amahi share that ''Uses pool'', then let Greyhole's nightly check move the files from there into the pool.
 
−
 
 
−
Here's a more detailed walkthrough for this method:
 
−
 
 
−
# Setup the Greyhole Storage Pool in the Amahi dashboard, Shares > Storage Pool page.
 
−
# Go in the Shares > Shares page. In there, make sure you have an existing share for each share you have data for. Create new ones if you need, remove the ones you don't need.
 
−
# Edit the ''path'' of each of those shares, and enter the current location of your existing data. For example, the Movies share could have a path = ''/media/External Drive/Movies''
 
−
# Enable the ''Uses pool'' option for each of your shares, and select the number of extra copies you'd like, if any.
 
−
# Now, you either need to wait for Greyhole's nightly check to start, or you can start it manually from a terminal, as root, using this command: greyhole --fsck
 
−
# Monitor /var/log/greyhole.log to see when the ''fsck'' operation is done.
 
−
# Once ''fsck'' is done, your data has now been moved into the Greyhole storage pool (in the drives you selected in Shares > Storage Pool). All that should be left in the previous location of your data (''/media/External Drive/Movies'' from the previous example) should be symbolic links pointing to the new file copies. If the previous location is just empty directories (no symlinks), do not panic. This is normal if your previous location is an NTFS or FAT partition (drive).
 
−
# Move all those directories / symlinks from there into the correct folders in /var/hda/files/share_name
 
−
# Back in the Amahi dashboard, edit the path of the shares once again, and put back ''/var/hda/files/share_name'' (i.e. the folders where you moved the symbolic links).
 
−
# If you used an NTFS or FAT partition for the previous location, you'll need another fsck to create the symlinks where they should be. Either wait for midnight, or launch it manually, from a command line, as root: ''greyhole --fsck''
 
−
 
 
−
You're done. All your existing data is now stored in the various drives included in your storage pool, and are accessible via the Samba shares you have defined in the Amahi dashboard.
 
  
 +
Once finished, it's important that you check to see the file is properly formatted.  To do so, do the following:
 +
cd /var/hda/platform/html/config
 +
echo 'require "yaml"; YAML::load(File.open("greyhole.yml"))' | irb
  
−
== Adding a new drive to your HDA and storage pool ==
+
If you do not see any errors, then your ready for the next step.  Otherwise, ensure you correct the errors before proceeding.
−
This wiki page discusses [[Adding_a_second_hard_drive_to_your_HDA|how to add a new drive to your HDA and to your greyhole storage pool.]]
 
  
−
== Drive Mounted as /media ==
+
For the changes to be effective, you will need to force Amahi to:
−
Including any drive mounted as /media/Something in your storage pool is usually a bad idea.<br/>
 
−
Those mounts are created by the ''gnome-automounter'', which requires you to be logged in into X (Gnome) to become available.<br/>
 
−
This will create issues with Greyhole, which expects drives to always be available, and will take action when some of them are missing.
 
  
−
Follow this guide to [[Adding a second hard drive to your HDA|permanently mount your drives]], before you include them in your storage pool.
+
:<u>Ubuntu</u><br />
 +
:Regenerate the greyhole.conf file by editing a share, clicking it's path, and just clicking the Save button without actually changing the path.
  
−
== Monitoring Greyhole ==
+
:<u>Fedora</u><br />
−
Sometimes you might want to monitor what Greyhole is doing, for example when writing data to your greyhole shares for the first time. Here are a few commands you can type in a terminal to follow along.
+
:Restart the Dashboard with any one of these alternatives as [[Open_terminal|root user]]:
 +
 +
:* touch /var/hda/platform/html/tmp/restart.txt
 +
:* systemctl restart httpd.service
 +
:* reboot
  
−
'''Scrolling view of total Greyhole operations queue:'''
+
:Then regenerate the greyhole set up by changing any setting in Shares or in Drive pooling.
−
{{Code|while [ 1 <nowiki>==</nowiki> 1 ]; do greyhole --view-queue <nowiki>|</nowiki> grep Total; sleep 60; done}}
+
<!--== Convert Greyhole from SQLite to MySQL ==
−
 
 
−
'''Alternate way to watch Greyhole operations queue:'''
 
−
{{Code| watch -d greyhole --view-queue }}
 
−
 
 
−
 
 
−
'''Scrolling log of what files Greyhole is working on right now:'''
 
−
{{Code|tail -f /var/log/greyhole.log}}
 
−
 
 
−
 
 
−
'''Amahi App for watching the greyhole log in your web browser:'''
 
−
 
 
−
[http://www.amahi.org/apps/greyhole-logmon Greyhole LogMon]
 
−
 
 
−
== Convert Greyhole from SQLite to MySQL ==
 
 
Greyhole is already using MySQL in Amahi 6.0
 
Greyhole is already using MySQL in Amahi 6.0
  
Line 155: Line 108:
 
''To Convert to MySQL, run as root:''
 
''To Convert to MySQL, run as root:''
  
−
{{Code|/usr/share/greyhole/db_migration-sqlite2mysql.sh}}
+
/usr/share/greyhole/db_migration-sqlite2mysql.sh
  
 
To see what DB engine is Greyhole using, do this:
 
To see what DB engine is Greyhole using, do this:
  
−
{{Code|grep db_engine /etc/greyhole.conf}}
+
grep db_engine /etc/greyhole.conf
−
 
+
-->
−
== Disable Greyhole ==
 
−
For those who do not use Greyhole, you can disable it.  This is based on the fact you never have used it by enabling 'Uses pool' on any share.  Recommend using extreme caution as this could have unpredictable results.
 
−
 
 
−
Perform the following steps as user '''root''':
 
  
−
{{Code|chkconfig greyhole off
+
== Tips and Best Practices ==
−
rm /etc/monit.d/greyhole.conf
 
−
service monit restart
 
−
service greyhole stop}}
 
  
−
'''DO NOT''' attempt to remove the Greyhole package as it is a dependency of the HDA software.  Doing so will break your HDA.
+
Refer to [[Greyhole_Tips|Greyhole Tips]] for additional guidance.
  
−
= Testing and Reliability =
+
== Testing and Reliability ==
  
 
Check out the [[Greyhole grinder]] to help make Greyhole rock solid.
 
Check out the [[Greyhole grinder]] to help make Greyhole rock solid.
  
−
= Reconnect the Greyhole storage pool =
+
== References ==
−
 
+
* Good [http://revxatlarge.blogspot.com/2011/04/restoring-deleted-files-in-greyhole-and.html article] on Greyhole with terms explained.
−
If you upgrade and/or reinstall Amahi and need to get your drives with data on them to the new system, follow this  [[Reconnect_existing_Greyhole_storage pool]] guide.
 
−
 
 
−
= Greyhole Troubleshooting =
 
−
 
 
−
Page: [[Greyhole troubleshooting]]
 
−
 
 
−
= Landing Zone Considerations =
 
−
 
 
−
The landing zone or LZ is the area where files arrive first before being distributed to their final destination. See details in the [[Greyhole landing zone]] page.
 
−
 
 
−
= Manually updating Greyhole for Amahi =
 
−
Page: [[Greyhole updating]]
 
−
 
 
−
= Emptying Greyhole Attic =
 
−
{{Code|greyhole --empty-attic}}
 
−
About the attic: The attic is used like a Recycle Bin, or Trash.
 
−
That means you'll need to manually empty it once in a while.
 
−
To do so, use the --empty-attic parameter (see above).
 
−
Another option is to create a 'Greyhole Attic' Samba share.
 
−
More details about that here: https://github.com/gboudreau/Greyhole/wiki/AboutTrash
 
−
 
 
−
= Moving drives/data out of Greyhole =
 
−
You may need to remove a drive from Greyhole and transfer the share files to another drive: [[Greyhole moving data out of the pool]]
 
  
−
= Changing the Mount Points of your Greyhole drives =
+
* [https://github.com/gboudreau/Greyhole/wiki Greyhole Wiki]:  Provides additional guidance and information.
−
[[Changing Greyhole Mount Points]]
 
  
−
= Reference =
+
* [https://github.com/gboudreau/Greyhole/wiki/MigrateFromWHS Windows Home Server (WHS) Drive Extender Migration Guide]:  Move from WHS to Amahi and Greyhole.
−
Good [http://revxatlarge.blogspot.com/2011/04/restoring-deleted-files-in-greyhole-and.html article] on Greyhole with terms explained.
+
<br />

Latest revision as of 01:30, 17 January 2017

What is Greyhole

Greyhole is Amahi's Storage pooling technology of choice. Storage Pooling combines the space of multiple disk drives and makes them look as if they were all part of a single pool of disk space. Greyhole was designed independently of Amahi, however, with feedback and great work, the Greyhole team made it suitable for integration with Amahi.

Specifically, Greyhole:

  • Combines the space from multiple drives into a single volume
  • Distributes files across all drives in the pool
  • Creates multiple copies of files in admin-specified shares


Recommend you consult what not to put in Greyhole shares as a precaution when determining which shares to select.

You can add additional drives to your HDA and prepare them for use in Greyhole by following adding a second hard drive to your HDA guidance or use the Amahi Disk Wizard (Amahi 8 or greater).

Warning!
You should NEVER change or delete files in the shared directories (/var/hda/files/* by default) directly on the HDA, using the terminal, or the Gnome file browser, for the shares for which you checked the Uses pool option in the Amahi dashboard. If you want to work with files on Greyhole shares on the HDA, you should Mount Shares Locally.


You should be safe reading existing files directly, or adding new files directly, as long as don't care that your new files will only be moved into the storage pool during the next check, which runs automatically at midnight (or manually using greyhole --fsck). Until then, your new files will be stored in your shared directories (normally /var/hda/files/share_name/).

Also, touching anything inside the gh directories that Greyhole creates at the root of your partitions is a recipe for disaster. We strongly discourage you from using the root partition in a drive pool.


NOTE: All commands are executed as root user (Fedora) or preceded with sudo (Ubuntu).

Amahi 7 or greater

Amahi application Greyhole UI provides this capability. Refer to the Greyhole UI Application Transition guide for details on installation and configuration.

NOTE: If you would still like to setup your Greyhole pool manually, please see Manual Greyhole Setup.

Amahi 6

You need to go in the Settings tab, and select Enable Advanced Settings. Without Advanced Settings enabled, you won't see the following page and options.

Storage Pool

 

Select the drives you want available for your storage pool.

Share Options

The next step is to select the share you want to replicate across the pool.

From the Shares tab, select the Shares sub-catagory. We chose the Pictures share for this tutorial.

 

Check the option for "Uses pool" and choose the number of drives to replicate this share. In this instance we have chosen to use all drives available to the greyhole pool.

NOTE: Greyhole is not a backup solution. If a file is removed, all copies are removed.

Advanced Options

You can further configure Greyhole by manually editing /var/hda/platform/html/config/greyhole.yml.

NOTE: Options duplicated in /etc/greyhole.conf will be overridden by grehole.yml.

To see what options are available, and what they do, refer to the sample greyhole.conf provided with Greyhole.

greyhole.yml is in in YAML format. Most of it should be easy enough to modify. The only exception would be the sticky_files (and optional sticky_into) options.

Here's an example of how those should appear:

To specify the following to Greyhole in greyhole.conf: 

sticky_files = Music/
 sticky_files = Videos/Movies/
     stick_into = /mnt/hdd1/gh
     stick_into = /mnt/hdd5/gh
 sticky_files = Backups/CrashPlan/
     stick_into = /mnt/hdd0/gh

one should specify this in the greyhole.yml file, where precise indentation matters: 

sticky_files: 
 - - Music/
 - - Videos/Movies/
   - - /mnt/hdd1/gh
     - /mnt/hdd5/gh
 - - Backups/CrashPlan/
   - - /mnt/hdd0/gh

Once finished, it's important that you check to see the file is properly formatted. To do so, do the following: 

cd /var/hda/platform/html/config
echo 'require "yaml"; YAML::load(File.open("greyhole.yml"))' | irb

If you do not see any errors, then your ready for the next step. Otherwise, ensure you correct the errors before proceeding.

For the changes to be effective, you will need to force Amahi to:

Ubuntu
Regenerate the greyhole.conf file by editing a share, clicking it's path, and just clicking the Save button without actually changing the path.
Fedora
Restart the Dashboard with any one of these alternatives as root user:
  • touch /var/hda/platform/html/tmp/restart.txt
  • systemctl restart httpd.service
  • reboot
Then regenerate the greyhole set up by changing any setting in Shares or in Drive pooling.

Tips and Best Practices

Refer to Greyhole Tips for additional guidance.

Testing and Reliability

Check out the Greyhole grinder to help make Greyhole rock solid.

References

  • Good article on Greyhole with terms explained.


Retrieved from "https://wiki.amahi.org/index.php?title=Greyhole&oldid=106326"