Upgrading SqueezeCenter to SqueezeBoxServer

Logitech has released an upgrade of SqueezeCenter (now called SqueezeBoxServer btw) and here are the steps for upgrading from version 7.3.2 to 7.4.1.

1. Uninstall SqueezeCenter 7.3.2

   # /ffp/sbin/funpkg -r squeezecenter-7.3.2-1.tgz
   

2. Upgrade and install additional necessary Perl packages
   

   # wget -r -l1 --no-parent -A.tgz http://www.inreto.de/dns323/fun-plug/0.5/extra-packages/perl/
   # /ffp/sbin/funpkg -u *.tgz
   # /ffp/sbin/funpkg -i *.tgz
   

3. Install SqueezeBoxServer 7.4.1

   # wget wget http://www.inreto.de/dns323/fun-plug/0.5/extra-packages/squeezecenter/squeezeboxserver-7.4.1-1.tgz
   # /ffp/sbin/funpkg -i squeezeboxserver-7.4.1-1.tgz

A couple of things to note:

1. SqueezeBoxServer installs to /ffp/lib/squeezeboxserver and /ffp/etc/squeezeboxserver.
2. You can delete any remainings of /ffp/lib/squeezecenter and /ffp/etc/squeezecenter.
3. Before starting squeezecenter, it's safest to reset the database using /ffp/sbin/reset-slimserver-database.sh. Note that all your previous settings are preserved but your music library will be rescanned.
   
4. Change the symlink for the logs in /ffp/var/log

   # rm /ffp/var/log/squeezecenter
   #ln -s /ffp/lib/squeezeboxserver/Logs /ffp/var/log/squeezeboxserver
   

Need to sell/swap/throw/RMA/give away your hard disk? Sanitize it first!

Six months after my first NAS capacity upgrade, I finally managed to upgrade the capacity of the second hard drive in my NAS to 1.5TB as well. I had actually wanted to do this since the Western Digital launched the Caviar Green model for this size but stock was hard to come by.

Replacing the old 500GB hard drive with the new 1.5TB one in the NAS is easy enough, especially given my last experience. What to do with the old drive became a problem. In the first upgrade, I re-purposed the old disk as an external backup drive for my folks by housing it in a USB HDD casing. 500GB for a pair of old folks who ain't really into digital content is more than enough hence I figured giving them a second external backup drive will probably be a waste. I won't be needing it too since my storage needs are served by my NAS. As such, I decided to sell it away on my favorite online marketplace.

Now if the item I'm selling is an MP3 player, a CPU or any other components for that matter, I wouldn't think twice about just placing an ad in the forum the very day I decide to sell it. However, this being a device that stores content, and more specifically sensitive personal content (Other than personal pictures and videos, I digitize and store all my paper documents such as bills and statements), I figured I better sanitize it first to avoid subsequent misuse of my personal information. Call me paranoid but hey, better safe than sorry right?

In terms of how to sanitize the disk, there is no safer way than to physically destroy it either by incinerating it or smashing it into pieces. However, that obviously isn't going to work if you plan to sell the disk away.

The next best option is to run some software that will securely delete all your data from the drive. A couple of tools that I know of and have tried personally are:

1. Darik's Boot and Nuke (DBAN) - a popular, open-source, linux-based self-contained boot disk that wipes any attached hard disks it detects. Incorporates a few wiping techniques such as Gutmann, DoD 5220-22.M standard, and zero/PRNG fill.


2. Eraser - a Windows based utility that implements most of the techniques in DBAN.


3. SDelete - a Windows based utility that uses core Windows APIs for its operations. Note that this utility can perform secure deletion of individual files/folders instead of the entire hard disk.
   


4. HDDErase - a DOS based utility that relies on the hard disk firmware internal erase functions rather than block writes. Works only with newer hard disks (manufactured after 2001) since the erase function was not part of the ATA specification before that.

My current favorite is number (4) HDDErase. The key reason being that the secure erase is done at the lowest level possible ensuring that the operation is comprehensive (i.e. it also erases certain sectors not accessible by applications) and effective (i.e. it does not depend on assumptions on how the Operating System as well as the underlying physical hard disks perform read/write operations).

The minor grievance I have on this solution is that the software has to run in a true DOS environment (the command prompt window in Windows does not qualify) and the developer has not included a bootable ISO image as a release option.

Having said that, you can download UBCD which is a bootable CD/USB image containing HDDErase and a whole lot of other PC diagnostics tools. Note that its about 100MB in size hence its really an overkill if all you want is HDDErase (which weighs in at less than 1MB!).

For me, I used BootFlashDOS which is a simple executable (i.e. no install required!) that formats your USB stick and create a bootable DOS environment. Note that you will need a Windows XP or 2003 server host to run the executable though.

[EDIT: found an alternative way of creating a boot CD that do not require Windows XP or 2003 server. Check it out here.]

On demand start/stop SqueezeCenter possible?

My main gripe with SqueezeCenter is the ridiculous amount of RAM (around 90MB) it requires to run. On the DNS-323 which has only 64MB of RAM, this means constant virtual memory swapping even when the process is idle (i.e. not serving any clients).

This is rather annoying as it means the disks in the NAS can no longer spin down even when there are no user accessing the box. It has also made redundant all my previous efforts in getting the disks to spin down as often as possible (e.g. moving ffp, mysql db and www pages to a USB stick).

To get back my desired behaviour, I have three options:

1. Upgrade to 128MB of physical RAM
2. Run Squeezed (a lightweight alternative) instead of SqueezeCenter
3. Start/Stop SqueezeCenter on demand

The first option is my preferred option as it should improve overall performance and not just for SqueezeCenter. However, it is extremely risky (changes of bricking your box is very high) as it involves some hardware modification and low level bios code hacking. Not recommended for the faint hearted and those under warranty. (And I just so happen to fall into both categories)

The second option seems like a good idea but it is really basic right now and lacks a lot of SqueezeCenter's features. Perhaps some time in the future I will reconsider this option.

The third option is a good compromise between having the full feature set of SqueezeCenter and getting my power-saving mode back when my SqueezeBox Boom is turned off. The only drawback is that there is a 2-3 minutes wait (for starting the monster up on the DNS-323) before the SqueezeBox Boom gets connected to SqueezeCenter when it is powered on.

I followed the instructions in Yannick's post and managed to get it working on my box with little trouble. I also made a number of changes to make it more robust, configurable and fit the ffp environment better. If you have followed my previous post on getting SqueezeCenter running on the DNS-323, you may find this approach more friendly.

In addition, I created a startup script for the standby daemon so that ffp will start the standby daemon rather than SqueezeCenter on boot/reboot. The assumption here is that I most likely do not need SqueezeCenter when I boot/reboot the box and I rather let a SqueezeBox client trigger the startup of SqueezeCenter than to have it unnecessarily load up my boot process and increase the boot time.

Here's a log of what I did to get this setup working in my environment:

1. Set up standby daemon script
   1. Download and install script (link)
      

      # cd /ffp
      # wget -O standby_daemon.tar.gz http://forum.dsmg600.info/attachment.php?item=497
      # tar -xzvf standby_daemon.tar.gz
      # rm standby_daemon.tar.gz

   2. Configure sudoers to allow user nobody to trigger the script without password. [IMPT: do not blindly copy the code below! replace {your_hostname} with your real hostname!]

      # cd /ffp/etc
      # chmod u+w sudoers
      # echo "nobody {your_hostname}= NOPASSWD: /ffp/sbin/slimserver_standby_daemon.pl" >> sudoers
      # chmod u-w sudoers

2. Set up SqueezeCenter plugin - Server Power Control
   1. Download and install plugin from here

      # cd /ffp/lib/squeezecenter/Plugins/
      # wget -O SrvrPowerCtrl.zip http://forums.slimdevices.com/attachment.php?attachmentid=7357&d=1241562529
      # unzip SrvrPowerCtrl.zip
      # rm SrvrPowerCtrl.zip

   2. Restart SqueezeCenter to load the newly installed plugin.
   3. Go to the settings page of the plugin (Settings>Plugins, look for Server Power Control and click on settings on the right) after SqueezeCenter has restarted
   4. Check the first two options (Shutdown and Shutdown to SqueezeNetwork) and enter "sudo /ffp/sbin/slimserver_standby_daemon.pl" in the textbox below it (both of them). Leave the rest of the options unchecked as we can't use them on the DNS-323.
   5. At the end of the settings dialog, select the option Monitor idle Players and select Shutdown server for the field Action to take after idle time. Optionally you can also setup the same in End of day action with a shorter idle time.
   6. Save the settings and you are done.
   
   
3. [Optional] Start standby daemon instead of SqueezeCenter on boot/reboot
   

   # cd /ffp/start
   # chmod a-x slimserver.sh
   # chmod a+x slimserver_standby_daemon.sh

4. Tips and Tricks
   1. The standby daemon outputs a log file called standby_daemon.log in SqueezeCenter's log directory (i.e. /ffp/lib/squeezecenter/Logs/). It records the timestamps of the shutdown and startup events which I find useful for tracking the uptimes as well as for debugging purposes.
   2. The default settings in the standby daemon script does not shutdown MySQL. (My instance of MySQL is used for other purposes hence I have to let it run independently). If you wish to have the script manage that, change the var $stop_mysql from 0 to 1. The script will, however, start MySQL if its not already running regardless of this setting. (Make sense right?!)
      

A couple of things still need to be sorted out in this setup:

1. Wakeup prior to alarm - Since SqueezeCenter is no longer running all the time, alarms that plays from SqueezeCenter will not work. Initial idea is to modify script to set up a cron job to wake SqueezeCenter a few minutes prior to the alarm being triggered. Note that this should not affect you if your alarms are set on SqueezeNetwork and you have configured Server Power Control to automatically switch you to SqueezeNetwork when SqueezeCenter is being shutdown.
   
2. Scheduled music library scans - Since SqueezeCenter is no longer running all the time, the scheduled daily scans may not be taking place anymore. Initial idea is to modify script to look for this setting and setup a cron job for it when the daemon is active.
   

I do not have any urgency to get these features working as I currently do not use them. I will probably get round to them if there is demand for it.

Setting up SqueezeCenter on DNS-323

One of the best features of the Squeezebox Boom (or any other SqueezeBox devices for that matter) is the ability to remotely control it as well as to stream your own music collection to it. But to do this, you will need to install and run a server software called the SqueezeCenter somewhere on your network. The ideal place to host it will be where you store your music library and in my case, that would be my SOHO NAS, the Dlink DNS-323.

Typically, the first question that comes to mind when you need to install and run a software is the OS platform and hardware requirements. The good news is that SqueezeCenter is written in Perl and stores its data in a MySQL database hence will run on any OS platform which supports those two dependencies; And that practically means all common platforms out there e.g. Mac, Linux and Windows.

In terms of hardware requirements, its alittle heavy but nothing a recent PC cannot handle. On the DNS-323 however, it still runs but performance does take quite a hit. In my setup, the memory footprint is around 90MB for SqueezeCenter alone plus another 20MB for MySQL and CPU utilization spikes to 100% often when using the web interface. Note that the DNS-323 has just 64MB of physical RAM and a 500MHz ARM CPU. Having said that, the web interface is still fully usable (no time outs) although you do feel the lagginess with most operations.

Luckily, the weak server hardware does not affect its streaming abilities and so far (er.. that's just a couple of hours usage since I got my box only yesterday), I have not experienced any issues on my SqueezeBox Boom. From my observations, the CPU hardly broke a sweat (>5% usage) when streaming mp3 format natively and the response on SqueezeBox Boom controls are pretty jiffy.

Here's what I did to get SqueezeCenter up and running on the DNS-323:

1. Get ffp 0.5 up and running (see wiki or my post);
2. Setup Pre-requisite 1 - MySQL
1. If you don't already have MySQL running, see this post;
2. Create a new database and an associated db user for SqueezeCenter (NOTE: db and user should be called slimserver!) :
   

   # mysql -u root -p
   Enter password: your_root_password
   
   mysql> create database slimserver;
   mysql> grant all on slimserver.* to slimserver identified by 'your_password';
   mysql> flush privileges;
   mysql> quit

3. Setup Pre-requisite 2 - Perl
   
1. Download and install all the Perl packages found here:

   # wget -r -l1 --no-parent -A.tgz http://www.inreto.de/dns323/fun-plug/0.5/extra-packages/perl/
   # funpkg -i *.tgz
   

4. Setup and configure SqueezeCenter
   1. Download and install SqueezeCenter package (squeezecenter-7.3.2-1.tgz);
      

      # wget http://www.inreto.de/dns323/fun-plug/0.5/extra-packages/squeezecenter/squeezecenter-7.3.2-1.tgz
      # funpkg -i squeezecenter-7.3.2-1.tgz

   2. Configure SqueezeCenter to start automatically when the box boots as well as to start it up now
      

      # cd /ffp/start
      # chmod a+x slimserver.sh
      # ./slimserver.sh start
      

   3. When SqueezeCenter starts up, it can take quite a few minutes before the web console becomes accessible. When it does become ready, you can access it via the URL http://[replace with your nas ip or hostname]:9000/
   4. Once you get into the web interface, the first thing you should do is to setup the path to your music collection. If your music collection is scattered all over, you will need create a "holding" directory and create the appropriate symlinks in there as SqueezeCenter only allows you to specify one location.
   5. Next trigger the scanning process to catalog your collection. Given the weak hardware of the DNS-323, the scanning process can take a long time to complete. (My 2000 odd files took 1+ hour to complete!)
      
5. SqueezeCenter Tips and Tricks
1. The location of the SqueezeCenter installation files (including plugins) are found in /ffp/lib/squeezecenter/.
   
2. The location of the SqueezeCenter configuration files (including plugins) are found in /ffp/etc/squeezecenter/.
3. The location of the SqueezeCenter log files are found in /ffp/lib/squeezecenter/Logs/. Since this isn't in line with where all my other logs go, I created a symlink to it at /ffp/var/log/squeezecenter/.
4. You can set SqueezeCenter to automatically rescan your music collection once a day via the Schedule Rescan function in Settings>Advanced>Rescan Music Library.
   
5. To reduce the memory footprint of SqueezeCenter, go to Settings>Plugins and uncheck all the plugins that you do not use. For me, I turned off the following plugins:
   1. Amazon CD Store
   2. Deezer
   3. Digital Inputs
   4. MP3tunes Music Locker
   5. MusicIP
   6. Pandora
   7. Rhapsody
   8. RSS News Ticker
   9. SIRIUS Internet Radio
   10. Slacker
   11. SlimTris
   Restart the slimserver process for the changes to take effect.
   
6. If you find the display only shows the time and does not show the date in the "off/Standby" state, it is most likely because of your font setting. Change it (Settings>Player>Display>Standby Font) to "Standard" to see the date information.


7. The slimserver startup script provided by fonz does not guard against firing multiple instances of the slimserver process. To fix this problem, include the following chunk of code at the top of the slimserver_start() function.
   

   
   # check if daemon is already running, if yes, do nothing
   if proc_status $command | grep -q "running as pid"; then
   echo "INFO: ${name} already running! Nothing to do."
   return
   fi
   # check if mysqld is already running, if not, prompt user and exit
   if proc_status "mysqld" | grep -q "not running"; then
   echo "ERROR: mysqld has to be running before ${name} can be started!"
   return
   fi
   

8. The pre-configured SqueezeCenter settings uses named pipes for accessing MySQL, if you are having problems like database time-outs; e.g. you can connect to the web console but find that the scanning process did not find all your music files, check in the scanner log (/mnt/usb/ffp/lib/squeezecenter/Logs/scanner.log) for the following error messages:
   

   Slim::Schema::forceCommit (1496) Warning: Couldn't commit transactions to DB: [DBD::mysql::db commit failed: MySQL server has gone away at /mnt/usb/ffp/lib/squeezecenter/Slim/Schema.pm line 1493.
   ]
   Slim::Schema::Storage::throw_exception (70) Error: DBI Connection failed: DBI connect('database=slimserver','slimserver',...) failed: Can't connect to local MySQL server through socket '/ffp/var/run/mysql/mysql.sock' (2) at /mnt/usb/ffp/lib/squeezecenter/CPAN/DBIx/Class/Storage/DBI.pm line 652

   If it exists, then you may want to change SqueezeCenter to connect to the DB via TCP/IP instead. To do that, in your server config file, change the value of dbsource to "dbi:mysql:hostname=localhost;port=3306".

Overall, I'm very pleased with the setup. The only gripe is that my HDDs no longer spin down when the NAS is idle due to the virtual memory swapping caused by the SqueezeCenter process (which still consumes some CPU even when no clients are active!). I contemplated putting the swap space on the USB stick where ffp lives to circumvent this but decided not to as I'm not confident that my relatively new USB stick can take that kind of abuse (flash drives have limited write cycles). Perhaps its time to go for the 128MB memory upgrade hack!

Setting up MySQL on DNS-323

It took me awhile to get round to this (more than a year apparently) but I finally managed to get it done over the weekend. The trigger was because I am going to get a SqueezeBox Boom and will be running the associated controller software called SqueezeCenter (which requires the MySQL database) off the DNS-323.

Compared to the last time I was looking at doing this, the installation and setup is now a breeze, all thanks to fonz for creating a funplug package for it:

1. Download the package from fonz's website
   

   # wget http://www.inreto.de/dns323/fun-plug/0.5/packages/mysql-5.0.67-2.tgz

2. Install the package

   # funpkg -i mysql-5.0.67-2.tgz

3. Setup the MySQL config file using the small version of the included examples (remember that the DNS-323 has limited resources)

   # cp -a /ffp/etc/examples/mysql/my-small.cnf /ffp/etc/my.cnf

4. Define a storage location for your database files in the config file. For me, I have it on my USB stick as my usage pattern is mostly reads and I want to my HDD to remain in spin down state longer. Add in my.cnf after [mysqld] : datadir = /ffp/var/mysql
5. Initialize MySQL

   # mysql_install_db

6. Now create a database and have fun!

   # mysql -u root -p
   Enter password: your_root_password
   
   mysql> create database testdb;
   mysql> grant all on testdbuser.* to testdb identified by 'your_password';
   mysql> flush privileges;
   mysql> quit

   
   
7. In case of errors, look up the log found in /ffp/var/mysql/[replace with your hostname].err

Besides using it with SqueezeCenter, I intend to also use it for some web-app projects (e.g. Joomla!). Stay tuned!

How to send email from scripts running on DNS-323

In the full installation of ffp 0.5, two email related utilities are installed by default. They are

1. mailx - a Mail User Agent which allows you to send and receive mail using command line parameters, and
   
2. esmtp - a Mail Transfer Agent which transfer mail from one computer to another

Unfortunately, they are not setup properly out-of-the-box hence you won't be able to use them right away. There also isn't any mention of how to do this on the wiki nor the package readme files.

Well, as it turned out, it wasn't that hard to get it up and running anyway which might explain the lack of documentation. But for the uninitiated, all you need is to create a config file called esmtprc in your /ffp/etc folder and input your mail provider details there. To learn more about how to configure the esmtprc file, take a look at this sample.

Then to send mail, use the mailx command syntax like this

> cat /bs/some_file.txt mailx -s "the email subject line" -r sender@somewhere.com recepient@somewhere.com

which sends the content of the file "some_file.txt" as the body of the message, or to send it as an attachment, use this

>mailx -s "this is your subject line" -a /bs/some_file.txt -r sender@somewhere.com recepient@somewhere.com

To use gmail as your mail provider, you may want to check out this forum post.

Upgrading Transmission to 1.42

Since I'm in the mood to upgrade, I figured I might as well do so for Transmission. More specifically, I want to upgrade so that I can get more fine grain control over the files to download in a torrent; You can do this via the CLI or a third party GUI client like this only. i.e. You cannot access this feature using the web interface (yet). Other things that have changed moving from v1.2 to the current v1.42 include the following:

Notably, the working directory is now /mnt/HD_a2/.transmission-daemon instead of just /mnt/HD_a2/.transmission. (since v1.3) If you are upgrading, you will need to move your existing directory manually if you want to resume your existing downloads. You may also need to amend your custom addon scripts (if any) such as my enhanced transmission script.

Another change (also since v1.3) is that Clutch no longer depends on a separate web server as one is already built in. This sucks as you now have another web server running on the box listening on its own port (default is 9091). Having said that, it does make installation and maintenance easier.

Lastly, access to Transmission is now restricted to local host by default (doesn't matter if you are using the CLI or web interface) (this change was introduced in v1.42). To allow access from other hosts, you will need to edit the rpc-whitelist parameter in the settings.json configuration file.

For an (almost) idiot-proof guide on setting up Trasmission v1.42 on the DNS-323, check out this guide by andy. More info can also be found in this forum thread.

Upgrading DNS-323 firmware to 1.06

I knew about the existence of the new firmware when I bumped up the capacity of the NAS a couple of weeks ago. However, I resisted the upgrade as none of the changes/fixes mentioned in the release notes (see below) interests me.

Just today, while troubleshooting slowness in browsing files on my NAS from my Vista laptop, I came across the news release for Samba 3.2 which states better integration and support Vista (and Server 2008!). Hoping that the new firmware will have incorporated this version (or later), I decided to give it a shot and upgraded the firmware tonight.

Like my previous upgrade, the process is straight forward and quick; just download the file, unpacked it into a temporary folder, and upload it to the DNS-323 using the web admin console.

After a mandatory reboot, my NAS is happily running v1.06. Unfortunately, Dlink has chosen not to touch the Linux kernel nor the Samba server hence they remain as version 2.6.12.6 and 3.0.24 respectively. Looks like if I want to give the new 3.2.x a spin, I'm going to have to go the route of replacing the stock Samba server with fonz's package!

DNS-323 Firmware 1.06 Release Notes
===========================================================

Feature Additions:
1. Supports 1.5 TB Hard Drives
2. USB port supports UPS monitoring
3. Add option to Manually or Automatically Rebuild Raid
4. UPnP AV supports Microsoft XBoX 360, Sony PlayStation 3
5. Support FTP over SSL/TLS
6. Add Unicode support to FTP Server
7. Email Alerts supports Gmail

Function Changes:
1. Turn off Fan at low system temperature
2. Disable UPnP AV Server and iTunes Server by default.
3. Remove static DNS from the DDNS server provider list
4. Remove the DDNS Timeout setting from GUI, and change the Status design. fixed timeout timer as 576 hours
5. Remove the Auto Refresh Timer option for UPnP AV and iTunes Server, instantly Auto Refresh by default.
6. More user friendly Time Settings GUI design

Bug fixes:
1. Fix BT downloaded files sometimes are unable to be deleted
2. UPnP AV support large file (over 4GB)
3. Fix 1st HDD unable to sleep
4. Fix HDD cannot hibernate if UPnP AV or iTunes Server is enabled
5. Time stamp of files are incorrect when DST is enabled

Transmission Remote GUI - feature rich desktop client for Transmission!

Out of the box, Transmission offers you two ways of accessing its features and functions; via the command-line interface (aka CLI) which is clumsy or via the web client (formally known as Clutch) which is basic. Now, there is a third and better way of accessing Transmission.... introducing Transmission Remote GUI!

This is a cross platform (Linux and Windows for now. Mac OS X on the way) desktop client with a look and feel similar to uTorrent. It offers a lot more control and status information on your torrents than the web client, not to mention fast too.

If you are a power user, this client is definitely for you! Note that to use it, you will need Transmission v1.40 and up. *Sigh*, looks like an upgrade is in order for me!

Upsizing NAS Storage from 500GB to 1.5TB

I have been planning to upgrade my NAS storage capacity after it crossed the 90% mark a couple of weeks ago. Just over a year ago, I went with a 500GB hard drive (1TB drives were available in the market then but at a premium) expecting it to last me for at least two years. I was pretty amazed to find myself consuming more than 400GB in just over a year.

On further analysis, I found out that most of the space was used up by my family's photo and video albums; It's not really surprising considering that my DSLR captures each frame at ten megapixels and my videos are captured in VGA quality at thirty frames a second and stored as mjpeg.

With that hindsight, I decided to go all out and get the largest drive on the market today for this upgrade. Other than size, the other two requirements I had were low power consumption and noise. This is because my NAS runs 24/7 and it sits in the living room where the constant clicking noises from the drives can get pretty annoying, especially late at night.

Based on today's market, my ideal choice is the Western Digital Caviar Green 1.5TB. Unfortunately, WD has yet to launch this model and the largest it offers is a 1TB model (WD10EADS). I would have waited for it if its due to launch within the next few months but I couldn't find any news of it online.

My next best choice is the Seagate Barracuda 7200.11 1.5TB model (ST31500341AS). Although the power consumption and noise performance isn't as fantastic as the WD Caviar Green range (reviews here, here and here), its price per GB ratio makes it a hard-to-resist offer; At $0.19/GB for the WD 1TB model vs $0.14/GB for the Seagate 1.5TB model, that is more than 20% cheaper! In addition, its also runs on slightly lower power and alot quieter(!) than my current pair of 7200.10 500GB drives. But do be careful when buying the Seagate drives as there are apparently some buggy production firmwares out there. Based on user feedback, it appears that those with firmware CC1G are ok.

As a compromise, I decided to replace my primary drive first while waiting to see if the WD Caviar Green 1.5TB model will turn up in the next couple of months and at what price point. FYI, I run a JBOD configuration with a full disk to disk backup setup (i.e. HD_a2 -> HD_b2) in my NAS hence it is possible for me to break the upgrade process into two phases. Obviously, I will have to stop backing up some folders due to the smaller backup drive.

The upgrade process was simple enough; I first swapped out the backup drive in the NAS with the new one, booted up the box and let the firmware detect and format the drive accordingly. After that, I copied the entire content in my primary drive to the new drive via ssh using the command cp -a /mnt/HD_a2 /mnt/HD_b2.

Once the copy is complete (be patient! It can take a long time to copy all 400+GB over!), shutdown the box, remove the old primary drive, move the new primary drive from the left drive bay to the right, put in the old backup drive in the left drive bay and reboot. Job done!

A couple of things to note:

1. The primary drive is in the right drive bay while the backup drive is in the left.
   
2. My DNS-323 firmware version is 1.05 and is able to detect the drive correctly. However, the "format new drive" dialog in the web admin console shows the drive having zero bytes. Just ignore that and continue to format the drive as normal. However if you are running a RAID configuration with your pair of 1.5TB drives, you will require the 1.06 firmware.
   
3. The formatting process can take quite awhile (more than 30mins!), just be patient and let it complete.
   

Cleanboot finally working on my DNS-323!

If you are running fun_plug on your DNS-323, you may notice that your partitions are no longer unmounted properly at shutdown or reboot. You have this problem if you start seeing this line in your dmesg output:

EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended

The solution to this problem is a utility called cleanboot. Unfortunately, since the first version, it has not worked for me and quite alot of other folks; instead of cleanly unmounting the partitions before shutting down or rebooting, it hangs the box. Bummer...

The good news is that a fellow forumer SilentException persisted and managed to fixed the utility for his box (forum post here). I just tried it and it works on mine too! Hooray! You may want to give it a shot and see if it works for you.

Upgrading ffp 0.5 and moving it to USB... again

Recently I have been getting some comments on my post on moving ffp to a USB stick asking for help as the instructions seemed to have become invalid. It was only then I found out that fonz has been updating ffp without changing the version number. Alot of changed since I last upgraded to ffp 0.5 and while there are instructions on how to upgrade to the latest version, I decided to play it safe and just do a fresh reinstall. Here is a log of what I did:

1. Backup the file /mnt/HD_a2/fun_plug and the directory /ffp. (use the sym link to make sure you are backing up your active configurations)
2. Download the latest packages (i.e. fun_plug and fun_plug.tgz) and put them in /mnt/HD_a2/.
3. Delete the folder /mnt/HD_a2/ffp if you have it and reboot.
4. Telnet into the box (default has no username/pswd), disable telnet and enable ssh. (i.e. do a chmod a-x telnetd.sh and chmod a+x sshd.sh)
5. Download and install all the core packages found here. After the download, install using the command funpkg -i *.tgz. Also run the command funpkg -u *.tgz to update existing packages. (i.e. those that came in the fun_plug.tgz package)
6. <-- At this point, we have a working stock ffp working off the hard disk partition /mnt/HD_a2. Now to update the box with extra packages we need -->
7. Restore the svn startup script from backup (it contains the path to my local SVN repository)
8. unrar
9. Transmission
10. Create a local fun_plug script (/ffp/etc/fun_plug.local) to set up some convenience shortcuts. (/hd1 for /mnt/HD_a2, /hd2 for /mnt/HD_b2 and /usb for /mnt/usb)
11. <-- Now that we have all the features working, lets move it to the USB stick using the method found here -->
12. First create the directory /mnt/HD_a2/.bootstrap and download the necessary files into it. (namely setup.sh and usb-storage.ko)
13. Edit setup.sh to make sure that the variable USBFFPPART refers to the right USB partition. If you have only 1 USB drive attached and it has only 1 partition, then this should be /dev/sdc1.
14. Reboot to get the USB partition up and running. (Check by running df and look for /mnt/usb)
15. Copy the entire /ffp folder into the USB partition /mnt/usb. (cp -a /mnt/HD_a2/ffp /mnt/usb/)
16. Create a dummy file .usb-ffp in the root of the USB partition. (touch /mnt/usb/.usb-ffp)
17. Reboot again to get ffp working off the USB partition. (If you need to troubleshoot, check the ffp log at /mnt/HD_a2/.bootstrap/ffp.log)

A couple of things to note:

• To have ffp running off the hard disk again, simply remove the file /mnt/usb/.usb-ffp and reboot. You may want to do this when you need to run e2fsck (i.e. file system check) on the partition.
• To be able to unmount the first hard drive partition (i.e. /mnt/HD_a2) say for disk maintainance, you will need to copy the USB module (i.e. usb-storage.ko) to VRAM (/lib/modules is a good place to put it) before loading it. You can do so by editing the script setup.sh.
  

Moving ffp to a USB stick

Ok, I finally got down to moving ffp to a cheap 1GB USB stick. The motivation for doing so are:

1. Power saving - When there are little to no activities in the box, I do not want the hard disks to spin up unnecessarily.
2. Clean HD_a2 file system - I still haven't figure out why cleanboot is not working on my box hence I rather risk a corrupted USB stick containing only ffp than my precious 500GB of real data.

Here are the steps I took to get it working on my box: [Edit: The instructions below is no longer valid for ffp 0.5 dated Aug 20th and later. Please look here for the latest instructions.]

1. Get a cheap 1GB USB stick and plug it into the USB port behind the device. (duh!)
2. Download the necessary module (i.e. usb-storage.ko) from here and put it in the folder /ffp/lib/modules/[kernel_version]/. Note that this is a new directory that you will need to create. (TIP: run 'uname -r' on the command line first to get the kernel version and then substitute it in the path above)
3. Test the compatibility of the USB stick by running the usb mounting script that came with the ffp v0.5 package.
1. Note: you may need to edit the script var usbdisk_dev if you have a second HDD as the default value is sdb1.
2. run the script using the command ". ./usbdisk.sh"
3. If everything is ok, you should see the message "Mounting /dev/sdc1 on /mnt/HD_a2/sdc1". You may also want to run df to confirm that it is properly mounted.
4. Now that you are sure your USB stick works in the DNS-323, its time to format the stick in ext2. To do this, unmount the stick and run "mke2fs /dev/sdc1".
5. Now, mount the new fs and copy the entire ffp folder to the stick. i.e. run "mount /dev/sdc1 /mnt/HD_a2/sdc1", followed by "cp -a /mnt/HD_a2/ffp /mnt/HD_a2/sdc1/".
6. When we get here, we are now ready to replace the stock fun_plug script with one that will run ffp from the USB stick. There are a couple of variants floating about in the forum. Personally, I like the one by SilentException best as it identifies the stick via its UUID hence making sure it grabs the right stick for our mount point even if you have multiple USB sticks attached to the box.
1. Backup your original fun_plug script. (This is always a good idea)
2. Grab the script from here and put it in the root directory of /mnt/HD_a2.
3. Locate your device's UUID by running the command blkid /dev/sdc1.
4. Copy the UUID and replace the value in the variable USB_DEV_UUID in the script.
5. You may also want to edit/remove the portion after #MS (but before # run commands!)
6. Reboot the box and when it comes back up, check the log /mnt/HD_a2/ffp.log to make sure that you are now running ffp off the USB stick instead of the HDD.

Note that for convenience, this script also defines the shortcuts /hd and /usb to refer to /mnt/HD_a2 and /mnt/USB respectively.

BitTorrent client showdown - Transmission vs mldonkey

I'm giving up mldonkey for transmission+clutch for the following reasons:

• Smaller memory footprint and less CPU intensive - this is especially important in a small embedded system like the DNS-323. For comparison sake, top reports 54% memory usage(sometimes growing to near 90%!) for mldonkey but only 7% for transmission (or 25% including lighttpd and php for clutch)
  
• BitTorrent protocol only - since I only use bittorrent protocol, I do not need such a heavy weight client like mldonkey (which includes other protocols like file download aka HTTP/FTP, edonkey, overnet, gnutella and fasttrack).
  
• Separation of UI from the core - unlike mldonkey (which offers telnet and http interface by default), the only built-in interface in transmission is the CLI (aka Command Line Interface) via transmission-remote. This again gives you the chance to reduce resource utilization by (1) not running the Clutch interface hence no web server required or (2) running Clutch in a shared web server on the box (unlike mldonkey which uses an exclusive built-in standalone web server)
  
• Efficiency of the client - after 3 days with both clients turned on downloading the same files, it seems that transmission is more efficient than mldonkey by achieving better sustained speeds and yet using less connections.

Transmission and Clutch setup guide here.

Enhanced transmission script for your DNS-323

I miss the ability to auto-stop/shutdown the bittorrent client after all downloads have completed hence I decided to port the old mldonkey script over. Here is a quick summary on what it does.

• Improved the robustness of the startup, shutdown and restart sequence.
• Added function to auto-stop transmission when there are no more active downloads.
• Added function to auto-shutdown the box when there are no more active downloads.
• Added function to configure the behavior when there are no more active downloads. Valid arguments are:
• set_stop_when_done - stops transmission when there are no more active downloads;
• set_off_when_done - shutdowns the box when there are no more active downloads;
• set_donothing_when_done - let transmission continue to run when there are no active downloads (i.e. seeding);
• Use the function softstop_status to check what is the current configuration option.

The default behavior is set_stop_when_done. Command syntax is . /ffp/start/transmission.sh [cmd]. Here is the script below.

#!/ffp/bin/sh

###################################################################
# author: Sidney Chong (bfg100k[at]gmail.com)                     #
# version: 0.1                                                    #
# date: 15/05/2008                                                #
#                                                                 #
# Version History                                                 #
# ---------------                                                 #
# v0.1 - expanded original startup script to be more robust and   #
#        include auto-stop or auto-shutdown the box when          #
#        downloads are completed.                                 #
#                                                                 #
###################################################################

if [ -z ${TRANSMISSION_HOME} ]; then
  #setup the ENV variables if not found
  #this can happen when running from crontab
  echo "Environment variables not found, including default fun_plug profile."
  . /ffp/etc/profile
fi

TMP_CRONTAB="/ffp/log/crontab.tmp"

name="transmission-daemon"
command="/ffp/bin/transmission-daemon"
user=nobody

transmission_start()
{
  #check if daemon is already running. if yes, do nothing
  if transmission_status  grep -q "is running"; then
    echo "INFO: ${name} already running! Nothing to do."
    return
  fi

  if [ ! -d $TRANSMISSION_HOME ]; then
    su $user -c "mkdir $TRANSMISSION_HOME"
  fi
  echo "Starting ${name}..."
  su $user -c "$command -f -v 2 1>$TRANSMISSION_HOME/$name.log 2>&1 &"
  sleep 1
  if [ -S $TRANSMISSION_HOME/daemon/socket ]; then
    chmod 0777 $TRANSMISSION_HOME/daemon/socket
  else
    echo "Could not find Transmission socket"
    return
  fi

  #this step configures what to do if there are no more active downloads
  #default (stop_when_done) is to stop transmission.
  #change to off_when_done if you want to shutdown the box
  #change to donothing_when_done if you want to let it run (e.g. for seeding)
  _transmission_setup_softstop stop_when_done

  echo "Transmission startup sequence completed. Check ${TRANSMISSION_HOME}/${name}.log for more details."
}

transmission_stop()
{
  if transmission_status  grep -q "is running"; then
    echo -n "Stopping ${name}... "
  if [ -x /ffp/bin/transmission-remote ]; then
   /ffp/bin/transmission-remote -q
  else
   /bin/kill -9 $(pidof $command)
  fi
    echo "done."
  else
    echo "INFO: ${name} not running! Nothing to stop."
  fi
  _transmission_setup_softstop
  echo "${name} stop sequence completed."
}

transmission_status()
{
 _pids=$(pidof $name)
 if test -n "$_pids"; then
  echo "$name is running"
 else
  echo "$name not running"
 fi
}

transmission_softstop_status()
{
  if crontab -l  grep -q 'softstop_off';
  then
    echo "DNS-323 will shutdown when there are no active downloads."
  else
    if crontab -l  grep -q 'softstop';
    then
      echo "${name} will stop when there are no active downloads."
    else
      echo "softstop has not been configured."
    fi
  fi
}

transmission_restart()
{
  transmission_stop
  echo "Waiting for old instance to shutdown before starting new daemon..."
  while transmission_status  grep -q "is running"
  do
    sleep 5
  done
  transmission_start
}

# this routine checks to see if transmission is still alive
# and if there are any downloads active.
# if no downloads are active, it attempts to stop transmission.
# if "off_when_done" option is specified, it will attempt to
# shutdown the DNS-323 as well.
_transmission_softstop()
{
  echo -n "`date`: Checking downloads... "
  if /ffp/bin/transmission-remote -l  grep -q 'downloading';
  then
    echo "downloads are still active!"
  else
    echo "no active downloads found!"
    transmission_stop
    if [ "${1}" = "off_when_done" ]; then
      echo "attempting to shutdown the DNS-323 now. Bye Bye!"
      touch /tmp/shutdown
    fi
  fi
}

# this routine sets up softstop as a cron job that runs every half hour
# to ADD the cron job, pass "stop_when_done" into the routine,
# to ADD the cron job with shutdown option, pass "off_when_done",
# any other values (or no value) means REMOVE by default
_transmission_setup_softstop()
{
  echo "Setting up softstop on crontab... "
  crontab -l > ${TMP_CRONTAB}
  sed -i -e '/transmission.sh softstop/d' ${TMP_CRONTAB}
  TMP_NAME=""
  case "$1" in
    stop_when_done)
      TMP_NAME="softstop"
      echo "INFO: This job will stop $name when all downloads are completed."
    ;;
    off_when_done)
      TMP_NAME="softstop_off"
      echo "INFO: This job will attempt to shutdown the DNS-323 when all downloads are completed."
    ;;
    *donothing_when_done)
      echo -n "Removing job from crontab... "
    ;;
  esac
  if [ "${TMP_NAME}" != "" ]; then
    # sanity check - do not allow adding any job if transmission is not already running
    if transmission_status  grep -q "is running"; then
      echo -n "Adding cron job to run every half hour... "
      echo -e "0,30 * * * * /ffp/start/transmission.sh ${TMP_NAME} >> /ffp/log/${name}.cron.log 2>&1" >> ${TMP_CRONTAB}
      sed -i -e '/^\s*$/d' ${TMP_CRONTAB}
    else
      echo "ERROR: ${name} is not running. NOT adding job to crontab."
    fi
  fi
  crontab ${TMP_CRONTAB}
  rm ${TMP_CRONTAB}
  echo "done."
}


case "$1" in
  stop)
    transmission_stop
    ;;
  softstop)
    _transmission_softstop
    ;;
  softstop_off)
    _transmission_softstop off_when_done
    ;;
  set_off_when_done)
    _transmission_setup_softstop off_when_done
    ;;
  set_stop_when_done)
    _transmission_setup_softstop stop_when_done
    ;;
  set_donothing_when_done)
    _transmission_setup_softstop donothing_when_done
    ;;
  softstop_status)
    transmission_softstop_status
    ;;
  restart)
    transmission_restart
    ;;
  status)
    transmission_status
    ;;
  start'')
    transmission_start
    ;;
  *)
    echo "Usage: $0 startstoprestartstatusset_off_when_doneset_stop_when_doneset_donothing_when_donesoftstop_status"
    ;;
esac

Setting up transmission and clutch

Setting up transmission+clutch is really simple if you have upgraded to fonz's fun_plug version 0.5. Here's a quick how-to.

Transmission (aka the core) setup

1. Grab the transmission package transmission-1.20-ffp0.5.tgz.
2. Grab the required package curl-7.18.1.tgz.
3. Install both using funpkg -i . (FYI, funpkg is found in the /ffp/sbin folder in case this is not setup on your path. This usually happens when you telnet/ssh in as a normal user and then do a su to root)
4. By default, transmission will create the directory /mnt/HD_a2/.transmission where it stores its working files and also your downloads. To change the location of this folder, edit the variable TRANSMISSION_HOME in the startup script /ffp/start/transmission.sh. To change the downloads folder, you will need to use the CLI or web interface (see below).
5. Start transmission by using the command . /ffp/start/transmission.sh start (note the dot in the beginning and the space immediately after).
6. Make the script /ffp/start/transmission.sh executable if you want it to auto start every time you boot. i.e. chmod a+x /ffp/start/transmission.sh.

For controlling the core, you have two options.

(1) Using the CLI via transmission-remote

No installation is required as this is bundled together with the core transmission package. For help on how to use, just type in the command without any arguments. Using the CLI also means that you have to first telnet into the box.

(2) Using the web interface by installing Clutch

Clutch is a web application that connects to the transmission core via the daemon socket file. As it is distributed as a web application package separately from transmission, you will need to go through some installation steps. Also, as it is just a web application package, you will need to install a web server (with PHP capabilities) on your box as a pre-requisite.

1. Assuming you have ffp v0.5 already, you will have lighttpd (i.e. the web server) installed (but not running yet) by default. To get the PHP capabilities, download the package php-5.2.6-1.tgz and run the funpkg -i command to install it.
2. Configure your web server by copying the configuration script from the examples directory. i.e. cp /ffp/etc/examples/lighttpd.conf-with-php lighttpd.conf
3. The web server assumes that your web document root is at /mnt/HD_a2/www and you will need to create the necessary directories manually. i.e. mkdir -p /mnt/HD_a2/www/logs /mnt/HD_a2/www/pages
4. Now that we got the pre-requisite setup completed, time to grab the clutch package Clutch-0.4.tar.gz.
5. Note that this is not a ffp package and you do not use funpkg -i to install it. Instead, you simply untar it into your web document root directory. i.e. tar zxf Clutch-0.4.tar.gz -C /mnt/HD_a2/www/pages
6. You may wish to change the path to the clutch package for URL easy access. i.e. mv /mnt/HD_a2/www/pages/Clutch-0.4 /mnt/HD_a2/www/pages/clutch
7. And finally, to allow clutch to find your transmission core, create the file clutch/remote/data/socket.txt and store the path to the transmission socket file in there. i.e. echo /mnt/HD_a2/.transmission/daemon/socket > /mnt/HD_a2/www/pages/clutch/remote/data/socket.txt
8. Once all the above setup is done, we are ready to start the web server. To do so run the command . /ffp/start/lighttpd.sh start.
9. Make the script /ffp/start/lighttpd.sh executable if you want it to auto start every time you boot. i.e. chmod a+x /ffp/start/lighttpd.sh.
10. To access the web interface, point your browser to http://[ip_address_of_your_box]:8080/clutch. Note that Clutch does not support Internet Explorer so you have to use Firefox, Opera or Safari instead.
11. Note that by default, anyone can access the BT client via the URL above. If you are paranoid over security or thinking of exposing it over the internet (so that you can control your BT downloads from office?), you may want to setup access control via directory ACL in lighttpd.

Upgrading to ffp 0.5

Since I'm in the mood to upgrade, I figured I might as well upgrade fonz's fun_plug to version 0.5. This can be considered a major upgrade as there are quite a few fundamental changes such as the directory structure, default shell and the default packages included. This means that all the addon packages you installed (or built) based on the older 0.4 version or below need to be reinstalled and/or reconfigured.

The upgrade from 0.4 to 0.5 is painless enough using the instructions found here. After the install, get rid of telnet and get ssh back up (note that the default ssh included is openssh, not dropbear). Instructions for this is here.

A couple of things to note:

First up, the new path to the fun_plug is now at /mnt/HD_a2/ffp. A symlink at /ffp allows you to move the binaries to say a USB flash to reduce disk access and also to allow a clean shutdown of the HD_a2 file system.

Next the default env variables such as BINDIR, FUNPLUGDIR are all gone. So if you have any scripts that reference them, you will need to get them changed (e.g. svnserve.sh and my nightly backup script)

The default installed packages can be found under /ffp/var/packages. Additional packages can be downloaded from here. To install them, follow the instructions here. (Note: substitute -u with -i for install and -r for remove).

Unrar will need to be installed again using this package.

mldonkey is not yet packaged for this version but moving it from the old is real easy; Just copy the file mlnet from /mnt/HD_a2/fun_plug.d/bin to /ffp/bin, copy the file mlnet.sh from /mnt/HD_a2/fun_plug.d/start to /ffp/start and edit it for the path changes.

Well, that about sums up my experience with this upgrade. Now to try out the various packages that are built for this version!

DNS-323 Firmware 1.05b28 is out!

The official 1.05 firmware for the DNS-323 is finally out over at the SG support website. The most significant change is the inclusion of a bittorrent client in the firmware! Not that it matters to me since I already use mldonkey but hey, its progress...

Anyways, the upgrade was uneventful (thank goodness for that!) and my data remains intact. It was a heart stopping ten minutes watching the progress bar inch towards completion as I had forgotten to pull out the backup drive from the device before I hit the enter key and a reformat would had been disastrous! (Note to self: time to get my remote backup going!)

Oh, one thing to note about flashing firmware, apparently, it will remove all your samba shares so you have to manually set them up all over again after the flash.

unrar for the DNS-323

The rar format is a popular archival cum compression format that is commonly used in online communities for file distribution. If you need to unpack such archives on your DNS-323, you are in luck. Follow the instructions below and you are all set.

1. Download the binaries for the arm architecture here.
2. Unpack it into fun_plug's bin directory (i.e. tar -xjvf unrar-arm.tar.bz2 $BINDIR/)
3. Run the command unrar x <archive> <path_to_extract> to start extracting!

MySQL on the DNS-323 possible? YES!

Finally! Someone took the effort to create a native compilation of mySQL for the DNS-323! See details here. Will give it a spin and file a field report next week when I'm back from my business trip in Auckland.

This is good news. Next up, Perl 5.8.3 support and we can then reposition the DNS-323 as a SOHO App/DB server! :)