Uploading Really Huge Files

[bluemicrobyte]

My Internet connection suffers from frequent disconnections. I have a 2 GB file that I need uploaded to my server, and SmartFTP just won't get it done with these disconnects happening every few hours. (It simply starts over each time this happens)

How would I go about getting this 2GB file on to my server? Is there an FTP program that can continue an upload after a disconnect?

[OboeGuru]

Your program should be resuming broken transfers where they left off, so my suspicion is that your server for some reason isn't allowing that to happen.

I think a more pertinent question is, what the heck are you uploading that's 2GB in size and has to remain at that size? I suggest archive-spanning it and telling the archive software to separate every couple hundred MB or so; large single-file sizes increases the risk of read/write errors. A spanned archive with each individual unit being only a couple hundred MB greatly reduces that risk and is a pretty common practice.

There's also the possibility that your server just won't allow a single file that size to be uploaded, so spanning it will remove that possibility as well.

[bluemicrobyte]

1) I've noticed SmartFTP resetting transfers before, even for 50 MB files if I was disconnected during the upload.

2) Hehe, it's an .iso file of a custom DVD I made that I'd like to let my users download. I've put it in a single .rar file, and I don't know if my users would know how to re-assemble it if I were to separate the .rar file into parts (because honestly I'm not completely sure how to re-assemble it myself :P)

3) I'm assuming my server allows it since it's paid hosting and I have 300 gigs of storage space.

[OboeGuru]

You must have auto-resume on file transfers disabled, then. Either that, or an extremely outdated version of the software that doesn't have the feature. I guarantee that SmartFTP should be resuming broken file transfers.
Auto reconnect and resume of broken transfers

It's quite simple, a spanned RAR consists of the following: *.rar (the file your users would open) and *.r00, *.r01, *.r02, etc. All of those together would contain your ISO in separate chunks.

You might want to check on that anyways, just because you have copious amounts of file space doesn't necessarily mean that it allows an individual file of any size to be uploaded via FTP. Check with your hosting service provider.

[bluemicrobyte]

Hmmm, well I'll let it run overnight and let you know what things look like tomorrow. I'm at 400/1.79 GB and it says 11.5 hours left =P

[Modred]

You might try using jigdo (http://atterer.net/jigdo/) for downloading. I don't know if he provides a binary for the file maker, but you can probably compile it from source with MinGW or Cygwin. You then feed in your .iso image and it will go through and map out all of the files inside.

Then you just upload the individual files, in directory structure like on the ISO image, and provide the .jigdo and .template files that the program creates. Then anyone can use jidgdo-lite to download the entire archive, and jidgo will fetch all the file individually and reconstruct the ISO.

This way, you aren't uploading the entire 2 GB in one fell swoop. I would look into some more on how to set this up at the moment, but I need to run out. Let me know if you're able to get it working.

[TimP]

If you're feeling particularly crafty, you can use split (http://en.wikipedia.org/wiki/Split_%28Unix%29) to split the large file into smaller files (as small as you want) and then upload them. Then, assuming it's a Unix-like webhost, you could write a PHP script that would do a one time re-assembly on their server. It's win-win, you get to upload small files and your users get to download one large file. Plus you can do it without shell access on their machine. I can provide some examples if you'd like.

[bluemicrobyte]

Blast! It got up to 1.1 GB overnight and I let it run -- it reached 1.4 GB before I got disconnected (and yes, it was a disconnect because my AIM went offline, I couldn't load pages, etc) and then it restarted. 24 MB / 1.8 GB ftw!

If you're feeling particularly crafty, you can use split (http://en.wikipedia.org/wiki/Split_%28Unix%29) to split the large file into smaller files (as small as you want) and then upload them. Then, assuming it's a Unix-like webhost, you could write a PHP script that would do a one time re-assembly on their server. It's win-win, you get to upload small files and your users get to download one large file. Plus you can do it without shell access on their machine. I can provide some examples if you'd like.

Hmmm, that sounds like it might work, I can use winrar to create multiple parts as .rar files -- can a PHP script re-assemble those? (and if so, how?)

edit: its a linux web host
edit2: the 1.4 GB file is still sitting on the server, probably as a "corrupt" file -- is there any way I could simply continue uploading to this partially uploaded file using SmartFTP?

[TimP]

You don't need WinRAR at all. You can use a program called split to split the files and you can optionally use bzip2 to compress them.

You can get split at CoreUtils for Windows (http://gnuwin32.sourceforge.net/packages/coreutils.htm) and bzip2 at bzip2 for Windows (http://gnuwin32.sourceforge.net/packages/bzip2.htm). They're both command-line tools so you need to run them through command prompt. (Make sure you download the binaries and dependencies for CoreUtils and the binaries for bzip2)

You can run split with split -b 10m youriso.iso

It will generate a bunch of files starting with xaa and going through as many letters of the alphabet as it needs. By default it uses two letter suffixes so try pick a size so it splits to under 676 pieces (10MB would work). Once it's done, you can run bzip2 with bzip2 x* as long as those split pieces are the only file that start with x in that folder.

Once you've done that, upload them into a new folder on your server. Your PHP script should look something like this:

<?php
system("bzip2 -d x*");
system("cat x* > youriso.iso");
?>

Before you do any of that, make sure you can run this simple script:

<?php
system("uname -a");
?>

If that fails, you're not quite out of luck, but it's considerably more difficult.

If you do happen to have shell access already, let me know because the only reason you need the PHP script is to pass shell commands and it's easier to do it directly in a terminal.

[bluemicrobyte]

It requires shell access though? I don't have that by default -- I can request access to it (and its free) but I'm not sure if I want to go through the application process for that :P

I'll let smartFTP retry for a few days and if it doesn't get it eventually I may have to use that method. Thanks for posting it :)

Edit: system("uname -a"); returns "Linux box155.bluehost.com 2.6.19-1_3.BHsmp #1 SMP Mon Jan 15 07:50:58 MST 2007 x86_64 x86_64 x86_64 GNU/Linux" on the server I'm trying to upload the file to.

[TimP]

It doesn't require shell access and since that sample script works then the entire process would work.

[bluemicrobyte]

Alright, we'll I've downloaded the CoreUtils "Complete package, except sources" setup and installed it, but I'm not quite sure what to do from there. I tried the command you gave me and I got "split is not recognized as a command"

[Modred]

Alright, we'll I've downloaded the CoreUtils "Complete package, except sources" setup and installed it, but I'm not quite sure what to do from there. I tried the command you gave me and I got "split is not recognized as a command"

I'm going to guess that the directory including the new tools isn't on your system path?

If you know the directory you installed the programs to (I'd suggest you open up Windows Explorer and find the actual files, so you can copy/paste the path from the address bar), then you can add them to your system path.


Open the System Properties in Control Panel
On the Advanced tab, click Environment Variables (it's at the bottom)
In System Variables, find Path, select it and click Edit
Add the path to the executable files you get from CoreUtils to the end of this list (separate it from previous directories with a semi-colon)


After that, launch a new command window and see if it works.

[bluemicrobyte]

Ah, I didn't know I had to do that =P

After messing with the syntax of adding the new path (didn't know there couldn't be spaces) it seems to be working! I'll post an update with results once this finally finishes and is uploaded.....

Also, if I don't compress the file parts before uploading, do I need to run the first line in the php code:

<?php
system("bzip2 -d x*");
system("cat x* > youriso.iso");
?>

Because it looks to me like the first line is for unzipping and the next line is for combining the parts.

[Modred]

After messing with the syntax of adding the new path (didn't know there couldn't be spaces)
Behold the glory of the quotation marks.... =p

Also, if I don't compress the file parts before uploading, do I need to run the first line in the php code:

<?php
system("bzip2 -d x*");
system("cat x* > youriso.iso");
?>

Because it looks to me like the first line is for unzipping and the next line is for combining the parts.

That's right. If you don't do the compression before uploading, you don't need to decompress after you upload. Compress one of the files from the split to see how much of a space gain it gives you. If the gain is noticeable, and remember that even a small space gain on a few megabyte file will pay off for several gigs, then you do better to compress first.

[TimP]

The benefits of compression depend on the types of files in the ISO. If it's a file format that's already compressed (just about every modern multimedia codec), then the benefits won't be worth the time required for compression. Compression can save a lot of space on text files, uncompressed images, video, sound, and in some cases, program binaries. I get the impression that this will be mostly video content, so you can probably skip compression safely.

[bluemicrobyte]

Yeah the video has been compressed by Adobe Encore, plus the .iso file itself has already been packed into a .rar file for users to download. The original iso was 2.03 GB or so and the .rar is 1.79 GB (which has been sliced into 100 MB chunks for uploading and is currently on it's way to the server now).

[TimP]

I wasn't figuring on you uploading a split RAR, I figured you'd upload the split original file. It's going to add an extra few steps if the chunks are a split RAR. (you'll need to install an un-RAR'er on their server)

[Neo]

Err. You could always get a trial-based FTP Program like FlashFXP and upload it that way. It has a specific option to "resume" files. And I'm pretty sure that BlueHost (your host?) should support that.

Smartftp is rather strange since it tends to hide features, not on purpose, but just meh.

Spliting the file up seems like an uneeded hassle. It also puts a strain on people who wish to download the file -- and your server. Imagine people used Download Managers to download each "split file" part.

Meh.

Resume, i thought, was a pretty standard feature for any FTP-Based File transfer nowadays.

-Neo

[Modred]

Spliting the file up seems like an uneeded hassle. It also puts a strain on people who wish to download the file -- and your server. Imagine people used Download Managers to download each "split file" part.

Meh.

-Neo

The scripts TimP gave put the split parts back together, so the users aren't downloading 50 parts, but the one large file. ;)

[bluemicrobyte]

I wasn't figuring on you uploading a split RAR, I figured you'd upload the split original file. It's going to add an extra few steps if the chunks are a split RAR. (you'll need to install an un-RAR'er on their server)

I intend to leave the end product as a .rar file for users to download. I'll try the assembly script now and post the results shortly.....

edit: Well, it appears to have worked, I'll have to download the whole thing and see if it's 100% intact =P (but at least the filesizes match up :P)

Thanks for all the help!