Tucopy Mirror Client -------------------- Prior to the installation, please be sure to read the README.cpan file as it contains important information regarding the install process. INSTALLATION: ------------- This file will provide you with a brief description on how to install and configure the Tucopy mirroring client. The Tucopy installation process requires that you have root access to the machine you are installing it on. It is also highly recommended that you create a separate user called tucopy for the mirroring client to run as. To create the tucopy user on your system, you must run the following command as root: Please note some machines might not have the adduser command. In that case please use your system's equivalent adduser tucopy passwd tucopy Once the tucopy user has been created and you have downloaded the client from http://content.tucows.com/tucopy.shtml, you must change ownership of the file to the tucopy user and move it into the /tmp directory. To do so, as root run: chown tucopy. tucopy*.tar.gz mv tucopy*.tar.gz /tmp You must now log into the system as the tucopy user and extract the client by running tar -zxvf /tmp/tucopy*.tar.gz Proceed into the newly created tucopy directory that was created in /tmp You will now need root access for the duration of the install. Either log in as root or use the 'su' command. Once you have root access, please execute the install.pl file by running: perl install.pl This will launch the Tucopy installer which will prompt you with a few questions about the layout of your system and which configuration options you would like to enable. If you select something by mistake, do not worry, you will be able to manually edit the Tucopy config files once the installation has completed. You might find our sample install logs useful. You can find them at: 1) For new mirrors: doc/sample_install_fresh.readme 2) For mirrors that already have the content: doc/sample_install_old_mirror.readme We will now go over the options presented to you by install.pl User to run tucopy as: ---------------------- This is the default user (tucopy) that the script will be ran as. Please do not set this script to run as root. Perl Location: -------------- This is the location of the perl binary, normally found in /usr/bin/perl If you are not sure where perl is located on your machine, you can run 'updatedb' followed by 'locate perl |grep bin' at the shell prompt. Local Content Root: ------------------- This is the directory where Tucopy will place all the downloaded content. This directory must correspond with your apache directory settings for the mirror. IE: If you place your content in /mirror then you must have the document root for the mirror set to /mirror/windows in the httpd.conf file. Also, the /mirror directory should already exist and be writable by the tucopy user. Do you want to use compression: ------------------------------- This will enable software compression for the downloaded files, which will make the files smaller, thus reducing download time and bandwidth cost. If you select to use compression it will search the machine for the Compress::Zlib perl module as shown below. Checking for Compress::Zlib: not found. Do you want to install Compress::Zlib now Y/N [Y]? If you choose to install Compress::Zlib, it will then attempt to launch CPAN and fetch and install Compress::Zlib. If it fails or you do not wish to use Compress::Zlib, you will be prompted to use gzip instead as shown below: Do you want to install Compress::Zlib now Y/N [Y]? n gzip location [/usr/bin/gzip]? If the Compress::Zlib installation fails, please see the README.cpan file for further instructions on how to install the Compress:Zlib perl module. Select Delivery Mechanism: LWP/wget/lynx: ----------------------------------------- We currently offer three different methods for fetching the tucows content. The most common and most efficient method is LWP. If for some reason you cannot use LWP, we suggest you use wget. Lynx should only be used as a last resort. Select delivery mechanism LWP/Wget/Lynx [LWP]? Checking for LWP:not found Do you want to install LWP now Y/N [Y]? If LWP is not already installed on your machine, the installer will launch CPAN and attempt to install LWP. If the install fails, it will expect you to install LWP manually (see the README file) after the install. If you select Wget you will be prompted with the following: (If you have it installed) Wget configuration: Wget location [/usr/bin/wget]? (If you do not have wget installed) which: no wget in (:/usr/X11R6/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/sbin:/bin:/usr/local/bin) Wget location []? You can specify /usr/bin/wget and then proceed to install wget after the tucopy installation is completed. If you select Lynx as the default you will see the following (if installed) Lynx configuration: Lynx location [/usr/bin/lynx]? Otherwise, like Wget, you will see the following and will be able to specify /usr/bin/lynx and then install lynx once the tucopy installation is complete. which: no lynx in (:/usr/X11R6/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/bin:/bin:/sbin:/bin:/usr/local/bin) Lynx location []? If any of the required perl modules fail to install, please see the README.cpan file for alternative ways to install them. Client root: ------------ This is the location in which you wish to install the Tucopy client code. For example, /usr/local/tucopy/tucopy Please note that /usr/local/tucopy must be writable by the tucopy user. You can create this directory as root and then use the chown command: mkdir /usr/local/tucopy chown tucopy. /usr/local/tucopy Dir doesn't exist, create it: ---------------------------- If you specified a directory for the Client root that does not exist, it will prompt you to allow the installer script to create it. Temp dir: /tmp -------------- This is the directory to use for temporary files such as locks Affiliate UserName: ------------------- This is your username that you use to log into Mirror Manager, located at http://content.tucows.com/login.html If you cannot 1remember your username, please use the "Forget Your Password?" feature at http://content.tucows.com/login.html If you are still having problems gaining your password, please contact affiliate.help@tucows.com and they will assist you. Affiliate Password: --------------- This is your password that you use to log into Mirror Manager. Please note that the username and password combination for the old Mirror.pl client (mirror/mirror) will not work with Tucopy. If you cannot remember your affiliate password, please use the "Forget Your Password?" feature at http://content.tucows.com/login.html If you are still having problems gaining your password, please contact affiliate.help@tucows.com and they will assist you. Max Download Speed: 0 for max ----------------------------- This is used if you would like to limit the amount of bandwidth the mirror script uses while it downloads the content. Setting this to a limit can save bandwidth costs if you have problems with bandwidth peaks. Speeds are measured in kilobytes. The Tucopy installation procedure will now recap the settings you have selected and once you press enter to continue, it will complete the installation. If you've selected a setting by mistake you can now edit the config and pkg files to modify your settings. The tucopy.cfg file is located in the client root/config directory (/usr/local/tucopy/config by default) and the package files are in the client root/config/packages/ directory. RUNNING TUCOPY: --------------- By default, Tucopy ships with .pkg files for each of the libraries we currently offer (bsd, linux, pda, windows, themes, games). If you are only mirroring Windows and Linux for example, you must remove the other .pkg files from the client root/config/packages directory (/usr/local/tucopy/config/packages by default) otherwise the Tucopy client will attempt to mirror those libraries and will receive authentication errors. If you are setting up a new mirror and NOT upgrading from mirror.pl, to run Tucopy for the first time, log into the tucopy account, change into the client root/bin directory (/usr/local/tucopy/bin by default) and run the following command: ./tucopy.sh -v This will start the Tucopy mirroring client and download the content required for each specific library. If you are running an existing mirror, and are upgrading FROM mirror.pl, you must now run the following command. ./tucopy.sh -v -sync This will update the timestamps on the files so you do not end up re-downloading the full library when you run Tucopy for the first time. After the initial -sync is done you must run: ./tucopy.sh -v Once that is done, your mirror will be updated. We recommend that you place the tucopy root/bin/tucopy.sh (/usr/local/tucopy/bin/tucopy.sh by default) command into your crontab to have it run automatically. To do so, run the following command as the tucopy user: crontab -e In your crontab, place the following lines: MAILTO=youremail@yourhost.com xx aa,bb,cc,.. * * * /usr/local/tucopy/bin/tucopy.sh You will need to replace xx with the minutes time (from 00 to 59) in which you want the script to run as and replace the "aa,bb,cc,.." with the list of hours you wish to run it at (00-23 format) (ie "xx aa,bb,cc,dd * * * /usr/local/tucopy/tucopy.sh" would run tucopy at aa:xx, bb:xx, cc:xx, dd:xx) You will also want to replace youremail@yourhost.com with a valid email address. An email will be sent to this address only if there are any errors while running Tucopy. For more information on cron, type man cron at the command prompt or try searching your favorite search engine with "/etc/crontab help". If the Tucopy client fails to fetch certain files, for example the HTML files, please run the client with the -force option. This will redownload the catalog files for each mirror, which contain listings of all the files and directories that should be in your mirror tree. Once you run tucopy.sh with the -force option, be sure to run it once more normally (tucopy.sh -v) and it will fetch the missing content. TUCOPY OPTIONS: --------------- Below are the different options Tucopy can be ran with: --help Shows the help file -sync Reads the package files, scans through all the files, and if file size matches, updates the timestamps. This will make sure that you will not have to re-download the whole mirror if you have a local copy that was obtained using some other mechanism. -force Downloads the catalog files for each mirror. This should be run once a month to ensure a clean mirror. -t This runs Tucopy in test mode, and will not update the mirror, it will only show what needs to be done. When using -t make sure you also use -v -v Verbose mode, which fully displays what Tucopy is doing. -V Prints the Tucopy version number. -c Allows the user to specify an alternative config file There are 3 files that you can use to check up on your mirror to verify that it is updating correctly. Those file are: .mirror_timestamp_a The last time Tucopy started updating .mirror_timestamp_z1 The last time Tucopy updated successfully .mirror_timestamp_z0 The last time Tucopy updated unsuccessfully These files can be found on your mirror at http://yourname.tucows.com/.mirror_timestamp_z1 etc You will see something like this: 987611506 YOURID "Windows" The first field is the unix time, the second field is your affiliate ID and the 3rd is the library. To transform the Unix time into a readable format, run the following: perl -e 'print scalar localtime(987611506);' If you do not have .mirror_timestamp_z0, do not panic, it simply means Tucopy has always updated your mirror successfully. LOGGING: -------- In the logs directory there are two log files, error_log and stats_log The error_log will contain any errors that occur while running this script, which can be used to help troubleshoot any problems you may be having. For more details on the error log, please see the error_log.readme The stats_log contains statistics on the mirroring process, such as the amount of files downloaded, the actual content size and the fetched size if you are using compression. If you have selected to use compression, which is recommended, it will also list the total savings vs not using compression. Included in your (client root)/bin directory (/usr/local/tucopy/bin by default) is a perl script called tustat.pl, which will parse the stats_log and output the statistics to your screen in a readable format. If you have specified /usr/local/tucopy as your client root, you can run tustat.pl by executing the following command: /usr/local/tucopy/bin/tustat.pl /usr/local/tucopy/logs/stats_log You will see something like this: ------------------------------------------------------------------------------ | Session | Package | Num. files | Content size | Fetched size | Savings | ------------------------------------------------------------------------------ | 20985 | LINUX | 10 | 58 | 58 | 0.00 %| | 20985 | Windows | 7 | 290 | 246 | 15.23 %| | 23846 | Windows | 32 | 4614 | 1115 | 75.82 %| ... ------------------------------------------------------------------------------ | Totals | | | | | | | | LINUX | 5156 | 962955 | 828613 | 13.95 %| | | Windows | 68333 | 14434397 | 13103165 | 9.22 %| ------------------------------------------------------------------------------ | All | | 73489 | 15397353 | 13931778 | 9.52 %| ------------------------------------------------------------------------------- Session This is the Session ID number. Package The library you are mirroring. Num. files The number of files fetched. Content size Actual size of the files fetched. Fetched size Total size of the fetched files with compression (if you are using compression). Savings Percentage of savings by using compression. For more information on the stats_log, please see the stats_log.readme CUSTOMIZATION: -------------- For the affiliates who wish to add their company logo to the Tucows mirrors, you will have to edit the get_ignore_pattern= field in *each* of the library package files (windows.pkg, linux.pkg etc) located in the config/packages directory. For example, if you have locallogo.gif and host.html, you will have to place them into the mirror content directory and then modify the following: get_ignore_pattern= to get_ignore_pattern=(^locallogo\.gif$)|(^host\.html$) For further information on adding your own logo to your mirror, please see the local_logo.readme located in the doc directory. If you require further help, please contact us at affiliate.help@tucows.com and we will be glad to help you.