Greyhole troubleshooting

Gathering Good Troubleshooting Information when reporting an issue with Greyhole, or asking for help regarding Greyhole. You should provide the following (execute the given commands as root).

1. What version of Fedora, Samba & Greyhole are you running?

bash code
​uname -r; rpm -q samba hda-greyhole​

2. The content of the /etc/samba/smb.conf & /etc/greyhole.conf files (provide paste URLs):

Fedora:

bash code
​yum -y install fpaste; fpaste /etc/samba/smb.conf fpaste /etc/greyhole.conf​

Ubuntu:

bash code
​pastebinit /etc/samba/smb.conf pastebinit /etc/greyhole.conf​

3. The result of the following commands:

bash code
​mount fdisk -l df -h greyhole --stats​

4. The list drives in your storage pool (per Amahi platform):

bash code
​mysql -u root -phda -e "select * from disk_pool_partitions" hda_production​

5. A list of the directories on the root of the drives included in your storage pool, obtained with the following command (provide a paste URL):

bash code
​mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production | grep -v 'concat(' | xargs ls -la | fpaste​

6. The Greyhole work queue:

bash code
​greyhole --view-queue​

If the above command returns nothing, execute the following to see the problem:

bash code
​tail /var/log/greyhole.log​

7. If you have issues with a particular file (file disappeared, is wrong size, has wrong filename), execute the following to find out what Greyhole did with this file:

bash code
​greyhole --debug filename​

Clearing The Greyhole Work Queue

I've not seen too many other reports of this occuring on the internet, so I suspect it's something that may be specific to my setup. However, the fix is pretty general, so I'm documenting it here for others to use.

Sometimes, greyhole will have too many tasks queued, and will lock up, leaving you with a full landing zone and an fsck queue that just isn't processing. Always leave your server overnight (or longer, depending on the size of your landing zone (overnight should be sufficient for most) to ensure that its not just taking a while to process, but if its really, definitely stuck, you may want to follow these instructions to clear the queue.

Stop the greyhole daemon

bash code
​sudo service greyhole stop​

Connect to the MYSQL engine

bash code
​mysql -u greyhole -pgreyhole​

Select the Greyhole database

bash code
​USE greyhole;​

Drop and Recreate the 'Tasks' table with the same structure

bash code
​TRUNCATE TABLE tasks;​

Exit the MYSQL engine

bash code
​exit;​

Restart the greyhole daemon

bash code
​sudo service greyhole start​

Schedule an fsck to clear out the files currently in the landing zone

bash code
​sudo greyhole --fsck​

Queue Still Stuck

If you've done the above, and all you've accomplished is another stuck fsck, it's time to get serious. The only solution I've found is to blow away the landing zone and then schedule the fsck with an empty landingzone. Greyhole will rebuild your file links from the files in your storage drives, so you won't lose anything except the data that was waiting to be moved to your storage drives. The steps below will mitigate even that loss.

Copy the contents of your landing zone somewhere else (preferably a non-greyhole share, so you can sort it out from the comfort and convenience of your normal workstation). In the example below, I just borrow the AFP share I have locally mounted for Apple timemachine backups, but this can be any folder. Don't copy the /drives subfolder, as that's actually the physical drives attached to the system.

bash code
​sudo cp -rv /var/hda/files/shareName /mnt/timemachine/landingzonebackup/shareName​

Now, stop the greyhole service again. Do this from the Amahi Console->Settings->Servers->Greyhole. Disable the watchdog and press the big red 'stop' button.

Blow away the contents of your landing zone shares. It is very important you don't touch the /drives folder within /var/hda/files if you have a default setup.

bash code
​sudo rm -rf /var/hda/files/shareName/*​

Now, clean up the horrible mess you'll probably have in each drive in /var/hda/files/drives - I had old shares and other assorted crap that was slowing down each fsck. Your /var/hda/files/drives/driveX/gh folder (replace X with whatever drive number you have) should only have references for your current shares. Use the code below to remove any folders that contain old shares.

bash code
​sudo rm -rf /var/hda/files/drives/drive/gh/folderName​

Then clean up the even more horrible mess you probably have in your /var/hda/files/drives/driveX/gh/.gh_graveyard folder. Note, this is a hidden folder, it won't be listed if you just ls the /var/hda/files/drives/driveX/gh folder. Use the command below to remove each folder that isn't a current share.

bash code
​sudo rm -rf /var/hda/files/drives/drive/gh/.gh_graveyard/folderName​

Now, restart the greyhole service using the Amahi Console. Remember to turn the watchdog back on.

Schedule another --fsck. It should start pretty much right away, and begin rebuilding the landing zone. Note that any shares you have won't be available again until the fsck completes.

bash code
​sudo greyhole --fsck​

Optional: You'll probably now want to check the landing zone backup you made, and recopy any actual data in the folder back in (don't just copy the whole folder back in, that would make everything you just did pointless).

Optional: Cleaning all the crap out of your physical drives has probably freed up a bit of space (or quite a bit, depending how much crap you have). Now is probably a good time to balance out your storage again.

bash code
​sudo greyhole --balance​

If you run into any trouble with the above, post it in the Amahi forums and PM me (U=doczombie) the link to your forum post if you like (PM's without a link to a forum post are liable to be ignored).

Database Error Table :greyhole.tasks does not exist

bash code
​ Can't describe tasks with query: DESCRIBE tasks - Error: Table 'greyhole.tasks' doesn't exist​

If the Greyhole service will not start, or you see this message when gathering info for support, it is possible that the Greyhole database was not created. The following MySQL commands will create the database:

bash code
​mysql -u root -phda -e 'create database greyhole' mysql -u root -phda -e "grant all on greyhole.* to greyhole@'localhost' identified by 'greyhole'" mysql -u greyhole -pgreyhole greyhole < /usr/share/greyhole/schema-mysql.sql​

Back to the Greyhole page