How to Transfer Large Files Over the Internet

by Don Singleton
Tulsa Computer Society
From the November, 2005 issue of the I/O Port Newsletter

How to Transfer Large Files Over the Internet

In our HelpingTulsa computer refurbishing project we have invited groups interested in learning how to do Computer Refurbishing to send someone to Tulsa for two or three days to learn how we do it, and then we give them a Road Warrior computer, which is the tool we use to transfer images to the machines being refurbished. We have helped a number of projects get started, and we were looking for a way to transfer a large file (from 0.5 gig to 2.5 gig) over the internet. I once tried uploading (with FTP) a 0.5 gig file, and then having one the groups try downloading it, and they kept getting errors. If I uploaded a GHO file, Ghost would say the file was not a good GHO file, and if I zipped it, WinZip would say it was a corrupted zip file. It is my opinion that FTP is not a robust enough protocol to transfer files that large. It certainly has some error checking and recovery, but with a file that large, there is a good chance that there will be double bit errors that will not be detected by their error detection check byte. A volunteer at one of the groups we had helped described the following procedure to me, and it worked, and so I wanted to document it in case any of you had a need to transfer very large files.

The first step is to ownload and install QuickPar, WinRar, and Free Download Manager. If you want more information about these files, here is where you can learn more about QuickPar, WinRar, and Free Download Manager.

WinRar

WinRar is a shareware file, and you can evaluate it for free for 40 days.

To purchase a copy that will run forever, visit the WinRar site to purchase a key for $29.00.

To set up WinRar click on Options and then Settings

Click on the Integration Tab, and select the file types you want WinRar to support. Definitely you want it to support RAR. I personally would not want it to support ZIP, since I use WinZip for that.

QuickPar

Install QuickPar and run it

And click Options

In the Integration section click Associate with PAR2 files and Associate with PAR files. In the File Deletion section click Delete Damaged Files after Repair and Send Files to Recycle Bin.


Creating Rar files

The first step is to use the WinRar program to break the large file(s) into a number of smaller files. In this example I have two files, one is approximately 2 gig, and the other is 0.5 gig that I want to transfer.

In a Desktop Explorer, highlight the file(s) to prepare. In this case rw3.GHO and rw300001.GHS), right click, and select Add to Archive

In the General Tab set Split to volumes to 15,000,000. This controls the size of the "rar" files, resulting in RAR files that are 14,649kb (nearly 15 meg). Set the other options shown in this screen capture, including checking Create SFX Archive, Put Recovery Record, and Test Archived Files, naming the Archive name to be whatever you want it to be. You can experiment with different values for the Split to volumes. A larger number will give you fewer files to have to download, but the larger they are, the more likely it will be that there will be errors introduced during the upload or download, and hence you may have to repeat the upload/download.

Then click on the Advanced tab.

Set the recovery record to 2 percent. Click OK to start the conversion. The following screen shows what you will have after three minutes (and eleven seconds). Three files have been created, and it is working on the fourth one, and it will take another 2 hours, 54 minutes, and 39 seconds to finish.

Time to go watch some TV.


Creating QuickPar files

When WinRAR finishes our folder contains 179 files, dan.part001.exe, and dan.part002.rar through dan.part179.rar. The next step is to use the QuickPar program to create a bunch of QuickPar files, which are "Parity Volumes" using the Reed Solomon algorithm which may be used to verify that a set of files have not been corrupted, or to reconstruct damaged files (providing that you have a sufficient quantity of Parity Volumes to match the missing or damaged files). Hilight all 179 files (hilight 1 of them, and press Ctrl A to hilight all of them, start the QuickPar program, and then drag and drop the files into the Source Files area in QuickPar

After you drag and drop the files, QuickPar will look like this. Note the arrow for Source Block Count. You can move this arrow from left to right to control the block count and resulting block size. For example, here you would have 2,498 blocks, and they would be 1,152,000 long. The next arrow, on Recovery Files gives us 250 blocks, and 21 recovery files.

Moving this second arrow we can have 63 recovery blocks, and 8 recovery files. Let us go with that, so click Create.

Initially it will test the files

And then it begins creating the first file dan.PAR2.

Here it is 13.1% through, and to should take about 10 minutes more.

dan.PAR2 completed in 14 minutes, 42 seconds. We said we wanted 8 recovery files, so it will take a while. Time to go watch some TV.

When it finished I had dan.par2, dan.vol00+01.PAR2, dan.vol01+02.PAR2, dan.vol03+04.PAR2, dan.vol07+05.PAR2. dan.vol12+09.PAR2. dan.vol21+14.PAR2, dan.vol35+14.PAR2, and dan.vol49+14.PAR2. That is 9 files. I am not sure I understand, since we specified 8 recovery files, but it works, so why sweat it.

Now I need to upload (using an FTP program) all 188 files (179 plus 9). It took over 12 hours (my Cox Cable has a fast download speed, but the upload speed is much slower), so I started the upload in the evening, and it finished about 10am the next morning.

To make it easy for someone to download the files I created a webpage with links to each file.



The files are now up,
so someone else can now download them.

Create an empty folder named RW, and download all of the following files to it. You will need to download 179 "part" files, and 9 other files. The first is dan.part001.exe, and the others are dan.part002.rar through dan.part179.rar. plus dan.par2 and dan.vol00+01.PAR2 through dan.vol49+14.PAR2

Before you faint at the idea of downloading 188 files one at a time, you can just download the 10 files in the first column, and then you can use Free Download Manager to get 178 files while you sleep. I am giving you links for all 188 files, just in case you find you need to download any of the 178 files again (if you find you got an error that could not be corrected on any of them), but hopefully you will not have to do that. Once you get the 10 files in the first column downloaded, click here.

dan.part001.exe
dan.par2
dan.vol00+01.PAR2
dan.vol01+02.PAR2
dan.vol03+04.PAR2
dan.vol07+05.PAR2
dan.vol12+09.PAR2
dan.vol21+14.PAR2
dan.vol35+14.PAR2
dan.vol49+14.PAR2


002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179



Using Free Download Manager to download 178 files in Batch Mode

To download the 178 files dan.part002.rar through dan.part179.rar run Free Download Manager.

Click Download

Click Create Batch Download

Fill in this information: (note, I am obscuring the actual URL, because the files are not there anymore, and they are not germane to this article anyway)

URLhttp://********/dan.part(*).rar
From:2
To:179
Wildcard bytes:3
Step:1
From:A
To:Z
Save To Folder:Whatever folder you want the files put in

This will give you:

If it does not start automatically, click the Green Right Arrow to start the download. In this image it is working on dan.part002.rar, a 14.3mb file, and it is now 23% done (3447 kb), and has another 2 min 24 seconds on that file.

Here we have finished dan.part002.rar and have started on dan.part003.rar

This is going to take a long time, so I suggest you go to bed, and leave it running.

It is actually possible to use Free Download Manager to download the other 10 files, but I think it is just as easy to download them one by one. If you have already downloaded the other 10 files, click here, otherwise if you want to use Free Download Manager to download them, then with it running go to your browser, and look at the list of files to be downloaded, right click one and select Save Target As (or Save Link As)

You will get something like this:

Click OK, and then repeat for the other 9 files.



Check files for errors

With a desktop explorer open on the folder containing the downloaded files, double click on dan.par2.

This will open QuickPar, and it will begin scanning the files. The blue bar will show progress on each file, but watch the bar below that. In this screenshot it has only processed 14 files, and it still needs to process 165 more (for a total of 2302 blocks).

Files that are OK will show up in green. If it finds a file missing, like dan.part170.rar, it will show up in red. If it finds a file has errors, like dan.part171.rar it will show up in orange. Once it finishes, the line will turn green.

If there are too many errors, it will show that it needs more blocks, and you will need to manually download the files with errors.

If it thinks it can recover any errors, it will show like this, and if you click Repair it will fix the errors.

Once you click Repair it will begin going through the 179 files. Here we are when we are just 3% through. 172 of the files are good, but we are going to see if we can repair the others (shown in orange).:

We finished. It took 5 minutes, 14 seconds, but if you check all 179 files are now green.

Once QuickPar shows all 179 files to be green, we can move on to using WinRar to convert them to the GHOST files we need. Using the Desktop Explorer, double click on dan.part001.exe. This will bring up WinRar. Set the Destination Folder to the folder you want the two files (rw3.GHO and rw300001.GHS) to be placed in. Then click Install

Here is a screen shot at the very start, when it is working on rw300001.GHS, and at this instant it has just processed two of the files.

The program is now finished.

And if we close WinRar and look at our Desktop Explorer we see

You should end up with two files: RW3.GHO (2,097,137kb) and rw300001.GHS (546,019kb)

This Road Warrior only has my CHILD16.GHO file (you will have to copy other GHO and GHS files from your old Road Warrior), but it has everything else, including an updated set of HowTo pages, updated drivers, and the current Customize Software feature.





For more information on the Tulsa Computer Society click here




Tulsa Computer Society 11/01/2005
Don Singleton, President