Difference between revisions of "Greyhole troubleshooting"

From Amahi Wiki
Jump to: navigation, search
 
(33 intermediate revisions by the same user not shown)
Line 1: Line 1:
Gathering Good Troubleshooting Information when reporting an issue with Greyhole, or asking for help regarding Greyhole.  You should provide the following (execute the given commands as root).
+
__TOC__
  
1. What version of Fedora, Samba & Greyhole are you running?
+
 
{{Code|uname -r; rpm -q samba amahi-greyhole}}
+
<b>NOTE:</b>  All Greyhole commands must be executed as ''root'' user or prefixed with <code>sudo</code>.
 +
 
 +
Gathering good troubleshooting information when reporting an issue with Greyhole, or asking for help.  You should provide the following in a text file or use fpaste or [https://pastebin.com/ Pastebin] (please '''DO NOT''' post excessive amounts of text content). 
 +
 
 +
Provide the info below to [https://greyhole.freshdesk.com/support/home Greyhole support on Freshdesk], not [https://forums.amahi.org Amahi Forums]:
 +
 
 +
1. What version of OS, Samba & Greyhole are you running?
 +
 
 +
<blockquote><u>Fedora:</u></blockquote>
 +
uname -r; rpm -q samba amahi-greyhole
 +
 
 +
<blockquote><u>Ubuntu:</u></blockquote>
 +
uname -r; dpkg -s samba greyhole
  
 
2. The content of the /etc/samba/smb.conf & /etc/greyhole.conf files (provide paste URLs):
 
2. The content of the /etc/samba/smb.conf & /etc/greyhole.conf files (provide paste URLs):
  
<u>Fedora:</u>
+
<blockquote><u>Fedora:</u></blockquote>
{{Code|yum -y install fpaste;  
+
yum -y install fpaste;  
fpaste /etc/samba/smb.conf  
+
fpaste /etc/samba/smb.conf  
fpaste /etc/greyhole.conf}}
+
fpaste /etc/greyhole.conf
 +
 
 +
::'''NOTE:'''  <code>yum</code> (deprecated) has been replaced with <code>dnf</code> as the package manager for Fedora 23 and greater.
  
<u>Ubuntu:</u>
+
<blockquote><u>Ubuntu:</u></blockquote>
{{Code|pastebinit /etc/samba/smb.conf  
+
pastebinit /etc/samba/smb.conf  
pastebinit /etc/greyhole.conf}}
+
pastebinit /etc/greyhole.conf
  
 
3. The result of the following commands:
 
3. The result of the following commands:
{{Code|mount
+
mount
fdisk -l
+
fdisk -l
df -h
+
df -h
greyhole --stats}}
+
greyhole --stats
  
 
4. The list drives in your storage pool (per Amahi platform):
 
4. The list drives in your storage pool (per Amahi platform):
{{Code|mysql -u root -phda -e "select * from disk_pool_partitions" hda_production}}
+
mysql -u root -p -e "select * from disk_pool_partitions" hda_production
 +
 
 +
::'''WARNING:'''  Please see [[Database Root Password]] for the appropriate login password.
  
 
5. A list of the directories on the root of the drives included in your storage pool, obtained with the following command (provide a paste URL):
 
5. A list of the directories on the root of the drives included in your storage pool, obtained with the following command (provide a paste URL):
{{Code|mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" hda_production <nowiki>|</nowiki> grep -v 'concat(' <nowiki>|</nowiki> xargs ls -la <nowiki>|</nowiki> fpaste}}
+
 
 +
<blockquote><u>Fedora</u></blockquote>
 +
mysql -u root -p -e "select concat(path, '/gh') from disk_pool_partitions" \
 +
hda_production | grep -v 'concat(' | xargs ls -la | fpaste
 +
 
 +
::'''WARNING:'''  Please see [[Database Root Password]] for the appropriate login password.
 +
 
 +
<blockquote><u>Ubuntu</u></blockquote>
 +
mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" \
 +
hda_production | grep -v 'concat(' | xargs ls -la | pastebinit
  
 
6. The Greyhole work queue:
 
6. The Greyhole work queue:
{{Code|greyhole --view-queue}}
+
greyhole --view-queue
If the above command returns nothing, execute the following to see the problem:
+
:If the above command returns nothing, execute the following to see the problem:
{{code|tail /var/log/greyhole.log}}
+
tail /var/log/greyhole.log
  
 
7. If you have issues with a particular file (file disappeared, is wrong size, has wrong filename), execute the following to find out what Greyhole did with this file:
 
7. If you have issues with a particular file (file disappeared, is wrong size, has wrong filename), execute the following to find out what Greyhole did with this file:
{{Code|greyhole --debug filename}}
+
greyhole --debug filename
  
== Clearing The Greyhole Work Queue ==
+
== Greyhole Work Queue ==
 
I've not seen too many other reports of this occuring on the internet, so I suspect it's something that may be specific to my setup. However, the fix is pretty general, so I'm documenting it here for others to use.
 
I've not seen too many other reports of this occuring on the internet, so I suspect it's something that may be specific to my setup. However, the fix is pretty general, so I'm documenting it here for others to use.
  
Line 41: Line 66:
  
 
Stop the greyhole daemon<br />
 
Stop the greyhole daemon<br />
<u>Fedora 19</u>
+
 
{{Code|systemctl stop amahi-greyhole.service}}
+
<blockquote><u>Fedora</u></blockquote>
<u>Ubuntu</u>
+
systemctl stop amahi-greyhole.service
{{Code|service greyhole stop}}
+
 
 +
<blockquote><u>Ubuntu</u></blockquote>
 +
service greyhole stop
 +
 
 
Connect to the MYSQL engine
 
Connect to the MYSQL engine
{{Code|mysql -u greyhole -pgreyhole}}
+
mysql -u greyhole -pgreyhole
 +
 
 
Select the Greyhole database
 
Select the Greyhole database
{{Code|USE greyhole;}}
+
USE greyhole;
 +
 
 
Drop and Recreate the 'Tasks' table with the same structure
 
Drop and Recreate the 'Tasks' table with the same structure
{{Code|TRUNCATE TABLE tasks;}}
+
TRUNCATE TABLE tasks;
 +
 
 
Exit the MYSQL engine
 
Exit the MYSQL engine
{{Code|exit;}}
+
exit;
 +
 
 
Restart the greyhole daemon<br />
 
Restart the greyhole daemon<br />
<u>Fedora 19</u>
+
 
{{Code|systemctl start amahi-greyhole.service}}
+
<blockquote><u>Fedora</u></blockquote>
<u>Ubuntu</u>
+
systemctl start amahi-greyhole.service
{{Code|service greyhole start}}
+
 
 +
<blockquote><u>Ubuntu</u></blockquote>
 +
service greyhole start
 +
 
 
Schedule an fsck to clear out the files currently in the landing zone
 
Schedule an fsck to clear out the files currently in the landing zone
{{Code|greyhole --fsck}}
+
greyhole --fsck
  
===Queue Still Stuck===
+
==Queue Still Stuck==
 
If you've done the above, and all you've accomplished is another stuck fsck, it's time to get serious. The only solution I've found is to blow away the landing zone and then schedule the fsck with an empty landingzone. Greyhole will rebuild your file links from the files in your storage drives, so you won't lose anything except the data that was waiting to be moved to your storage drives. The steps below will mitigate even that loss.
 
If you've done the above, and all you've accomplished is another stuck fsck, it's time to get serious. The only solution I've found is to blow away the landing zone and then schedule the fsck with an empty landingzone. Greyhole will rebuild your file links from the files in your storage drives, so you won't lose anything except the data that was waiting to be moved to your storage drives. The steps below will mitigate even that loss.
  
 
Copy the contents of your landing zone somewhere else (preferably a non-greyhole share, so you can sort it out from the comfort and convenience of your normal workstation). In the example below, I just borrow the AFP share I have locally mounted for Apple timemachine backups, but this can be any folder. Don't copy the /drives subfolder, as that's actually the physical drives attached to the system.
 
Copy the contents of your landing zone somewhere else (preferably a non-greyhole share, so you can sort it out from the comfort and convenience of your normal workstation). In the example below, I just borrow the AFP share I have locally mounted for Apple timemachine backups, but this can be any folder. Don't copy the /drives subfolder, as that's actually the physical drives attached to the system.
{{Code|sudo cp -rv /var/hda/files/shareName /mnt/timemachine/landingzonebackup/shareName}}
+
cp -rv /var/hda/files/shareName /mnt/timemachine/landingzonebackup/shareName
  
 
Now, stop the greyhole service again. Do this from the Amahi Console->Settings->Servers->Greyhole. Disable the watchdog and press the big red 'stop' button.
 
Now, stop the greyhole service again. Do this from the Amahi Console->Settings->Servers->Greyhole. Disable the watchdog and press the big red 'stop' button.
  
 
Blow away the contents of your landing zone shares. It is '''very''' important you don't touch the /drives folder within /var/hda/files if you have a default setup.
 
Blow away the contents of your landing zone shares. It is '''very''' important you don't touch the /drives folder within /var/hda/files if you have a default setup.
{{Code|rm -rf /var/hda/files/shareName/*}}
+
rm -rf /var/hda/files/shareName/*
  
 
Now, clean up the horrible mess you'll probably have in each drive in /var/hda/files/drives - I had old shares and other assorted crap that was slowing down each fsck. Your /var/hda/files/drives/driveX/gh folder (replace X with whatever drive number you have) should ''only'' have references for your current shares. Use the code below to remove any folders that contain old shares.
 
Now, clean up the horrible mess you'll probably have in each drive in /var/hda/files/drives - I had old shares and other assorted crap that was slowing down each fsck. Your /var/hda/files/drives/driveX/gh folder (replace X with whatever drive number you have) should ''only'' have references for your current shares. Use the code below to remove any folders that contain old shares.
{{Code|rm -rf /var/hda/files/drives/drive/gh/folderName}}
+
rm -rf /var/hda/files/drives/drive/gh/folderName
  
 
Then clean up the even more horrible mess you probably have in your /var/hda/files/drives/driveX/gh/.gh_graveyard folder. ''Note, this is a hidden folder, it won't be listed if you just ls the /var/hda/files/drives/driveX/gh folder.'' Use the command below to remove each folder that isn't a current share.
 
Then clean up the even more horrible mess you probably have in your /var/hda/files/drives/driveX/gh/.gh_graveyard folder. ''Note, this is a hidden folder, it won't be listed if you just ls the /var/hda/files/drives/driveX/gh folder.'' Use the command below to remove each folder that isn't a current share.
{{Code|rm -rf /var/hda/files/drives/drive/gh/.gh_graveyard/folderName}}
+
rm -rf /var/hda/files/drives/drive/gh/.gh_graveyard/folderName
  
 
Now, restart the greyhole service using the Amahi Console. Remember to turn the watchdog back on.
 
Now, restart the greyhole service using the Amahi Console. Remember to turn the watchdog back on.
  
 
Schedule another --fsck. It should start pretty much right away, and begin rebuilding the landing zone. Note that any shares you have won't be available again until the fsck completes.
 
Schedule another --fsck. It should start pretty much right away, and begin rebuilding the landing zone. Note that any shares you have won't be available again until the fsck completes.
{{Code|sgreyhole --fsck}}
+
greyhole --fsck
  
 
'''Optional''': You'll probably now want to check the landing zone backup you made, and recopy any actual data in the folder back in (don't just copy the whole folder back in, that would make everything you just did pointless).
 
'''Optional''': You'll probably now want to check the landing zone backup you made, and recopy any actual data in the folder back in (don't just copy the whole folder back in, that would make everything you just did pointless).
  
 
'''Optional''': Cleaning all the crap out of your physical drives has probably freed up a bit of space (or quite a bit, depending how much crap you have). Now is probably a good time to balance out your storage again.
 
'''Optional''': Cleaning all the crap out of your physical drives has probably freed up a bit of space (or quite a bit, depending how much crap you have). Now is probably a good time to balance out your storage again.
{{Code|greyhole --balance}}
+
greyhole --balance
  
 
If you run into any trouble with the above, post it in the [http://forums.amahi.org/viewforum.php?f=39&sid=f676fa2944ab7d2ccfc8901ee731b6fe Amahi forums] '''and''' PM me (U=doczombie) the link to your forum post if you like (PM's without a link to a forum post are liable to be ignored).
 
If you run into any trouble with the above, post it in the [http://forums.amahi.org/viewforum.php?f=39&sid=f676fa2944ab7d2ccfc8901ee731b6fe Amahi forums] '''and''' PM me (U=doczombie) the link to your forum post if you like (PM's without a link to a forum post are liable to be ignored).
  
===Database Error Table :greyhole.tasks does not exist ===
+
==Database Table Error ==
{{Code| Can't describe tasks with query: DESCRIBE tasks - Error: Table 'greyhole.tasks' doesn't exist}}
+
Can't describe tasks with query: DESCRIBE tasks - Error: Table 'greyhole.tasks' doesn't exist
If the Greyhole service will not start, or you see this message when gathering info for support, it is possible that the Greyhole database was not created.  The following MySQL commands will create the database:
+
 
{{Code|mysql -u root -phda -e 'create database greyhole'
+
If the Greyhole service will not start, or you see this message when gathering info for support, it is possible that the Greyhole database was not created.   
mysql -u root -phda -e "grant all on greyhole.* to greyhole@'localhost' identified by 'greyhole'"
+
 
mysql -u greyhole -pgreyhole greyhole < /usr/share/greyhole/schema-mysql.sql
+
The following command will create the database:
service greyhole restart}}
+
 
 +
hda-create-db-and-user greyhole
 +
mysql -ugreyhole -pgreyhole greyhole < /usr/share/greyhole/schema-mysql.sql
 +
 
 +
Restart the greyhole daemon<br />
 +
 
 +
<blockquote><u>Fedora</u></blockquote>
 +
systemctl start amahi-greyhole.service
 +
 
 +
<blockquote><u>Ubuntu</u></blockquote>
 +
service greyhole start
 +
 
 +
==Drive is OFFLINE==
 +
Check that the ''/gh'' folder is created on the drive.
 +
 
 +
== Out Of iNodes ==
 +
If you run out of iNodes due to the size of the Greyhole Spool directory, the following should fix the problem:
 +
mv /var/spool/greyhole /var/spool/greyhole.bak
 +
mkdir -p /var/spool/greyhole
 +
chmod 777 /var/spool/greyhole
 +
/usr/bin/greyhole --create-mem-spool
 +
systemctl restart amahi-greyhole.service
 +
greyhole -f
 +
 
 +
Reference: [https://github.com/gboudreau/Greyhole/issues/123 Greyhole Issue 123]
 +
 
  
Back to the [[Greyhole]] page
+
Back to the [[Troubleshooting]].

Latest revision as of 22:12, 26 December 2020


NOTE: All Greyhole commands must be executed as root user or prefixed with sudo.

Gathering good troubleshooting information when reporting an issue with Greyhole, or asking for help. You should provide the following in a text file or use fpaste or Pastebin (please DO NOT post excessive amounts of text content).

Provide the info below to Greyhole support on Freshdesk, not Amahi Forums:

1. What version of OS, Samba & Greyhole are you running?

Fedora:

uname -r; rpm -q samba amahi-greyhole

Ubuntu:

uname -r; dpkg -s samba greyhole

2. The content of the /etc/samba/smb.conf & /etc/greyhole.conf files (provide paste URLs):

Fedora:

yum -y install fpaste; 
fpaste /etc/samba/smb.conf 
fpaste /etc/greyhole.conf
NOTE: yum (deprecated) has been replaced with dnf as the package manager for Fedora 23 and greater.

Ubuntu:

pastebinit /etc/samba/smb.conf 
pastebinit /etc/greyhole.conf

3. The result of the following commands:

mount
fdisk -l
df -h
greyhole --stats

4. The list drives in your storage pool (per Amahi platform):

mysql -u root -p -e "select * from disk_pool_partitions" hda_production
WARNING: Please see Database Root Password for the appropriate login password.

5. A list of the directories on the root of the drives included in your storage pool, obtained with the following command (provide a paste URL):

Fedora

mysql -u root -p -e "select concat(path, '/gh') from disk_pool_partitions" \
hda_production | grep -v 'concat(' | xargs ls -la | fpaste
WARNING: Please see Database Root Password for the appropriate login password.

Ubuntu

mysql -u root -phda -e "select concat(path, '/gh') from disk_pool_partitions" \
hda_production | grep -v 'concat(' | xargs ls -la | pastebinit 

6. The Greyhole work queue:

greyhole --view-queue
If the above command returns nothing, execute the following to see the problem:
tail /var/log/greyhole.log

7. If you have issues with a particular file (file disappeared, is wrong size, has wrong filename), execute the following to find out what Greyhole did with this file:

greyhole --debug filename

Greyhole Work Queue

I've not seen too many other reports of this occuring on the internet, so I suspect it's something that may be specific to my setup. However, the fix is pretty general, so I'm documenting it here for others to use.

Sometimes, greyhole will have too many tasks queued, and will lock up, leaving you with a full landing zone and an fsck queue that just isn't processing. Always leave your server overnight (or longer, depending on the size of your landing zone (overnight should be sufficient for most) to ensure that its not just taking a while to process, but if its really, definitely stuck, you may want to follow these instructions to clear the queue.

Stop the greyhole daemon

Fedora

systemctl stop amahi-greyhole.service

Ubuntu

service greyhole stop

Connect to the MYSQL engine

mysql -u greyhole -pgreyhole

Select the Greyhole database

USE greyhole;

Drop and Recreate the 'Tasks' table with the same structure

TRUNCATE TABLE tasks;

Exit the MYSQL engine

exit;

Restart the greyhole daemon

Fedora

systemctl start amahi-greyhole.service

Ubuntu

service greyhole start

Schedule an fsck to clear out the files currently in the landing zone

greyhole --fsck

Queue Still Stuck

If you've done the above, and all you've accomplished is another stuck fsck, it's time to get serious. The only solution I've found is to blow away the landing zone and then schedule the fsck with an empty landingzone. Greyhole will rebuild your file links from the files in your storage drives, so you won't lose anything except the data that was waiting to be moved to your storage drives. The steps below will mitigate even that loss.

Copy the contents of your landing zone somewhere else (preferably a non-greyhole share, so you can sort it out from the comfort and convenience of your normal workstation). In the example below, I just borrow the AFP share I have locally mounted for Apple timemachine backups, but this can be any folder. Don't copy the /drives subfolder, as that's actually the physical drives attached to the system.

cp -rv /var/hda/files/shareName /mnt/timemachine/landingzonebackup/shareName

Now, stop the greyhole service again. Do this from the Amahi Console->Settings->Servers->Greyhole. Disable the watchdog and press the big red 'stop' button.

Blow away the contents of your landing zone shares. It is very important you don't touch the /drives folder within /var/hda/files if you have a default setup.

rm -rf /var/hda/files/shareName/*

Now, clean up the horrible mess you'll probably have in each drive in /var/hda/files/drives - I had old shares and other assorted crap that was slowing down each fsck. Your /var/hda/files/drives/driveX/gh folder (replace X with whatever drive number you have) should only have references for your current shares. Use the code below to remove any folders that contain old shares.

rm -rf /var/hda/files/drives/drive/gh/folderName

Then clean up the even more horrible mess you probably have in your /var/hda/files/drives/driveX/gh/.gh_graveyard folder. Note, this is a hidden folder, it won't be listed if you just ls the /var/hda/files/drives/driveX/gh folder. Use the command below to remove each folder that isn't a current share.

rm -rf /var/hda/files/drives/drive/gh/.gh_graveyard/folderName

Now, restart the greyhole service using the Amahi Console. Remember to turn the watchdog back on.

Schedule another --fsck. It should start pretty much right away, and begin rebuilding the landing zone. Note that any shares you have won't be available again until the fsck completes. greyhole --fsck

Optional: You'll probably now want to check the landing zone backup you made, and recopy any actual data in the folder back in (don't just copy the whole folder back in, that would make everything you just did pointless).

Optional: Cleaning all the crap out of your physical drives has probably freed up a bit of space (or quite a bit, depending how much crap you have). Now is probably a good time to balance out your storage again. greyhole --balance

If you run into any trouble with the above, post it in the Amahi forums and PM me (U=doczombie) the link to your forum post if you like (PM's without a link to a forum post are liable to be ignored).

Database Table Error

Can't describe tasks with query: DESCRIBE tasks - Error: Table 'greyhole.tasks' doesn't exist

If the Greyhole service will not start, or you see this message when gathering info for support, it is possible that the Greyhole database was not created.

The following command will create the database:

hda-create-db-and-user greyhole
mysql -ugreyhole -pgreyhole greyhole < /usr/share/greyhole/schema-mysql.sql

Restart the greyhole daemon

Fedora

systemctl start amahi-greyhole.service

Ubuntu

service greyhole start

Drive is OFFLINE

Check that the /gh folder is created on the drive.

Out Of iNodes

If you run out of iNodes due to the size of the Greyhole Spool directory, the following should fix the problem:

mv /var/spool/greyhole /var/spool/greyhole.bak
mkdir -p /var/spool/greyhole
chmod 777 /var/spool/greyhole
/usr/bin/greyhole --create-mem-spool
systemctl restart amahi-greyhole.service
greyhole -f

Reference: Greyhole Issue 123


Back to the Troubleshooting.