Hi all,
I've been having the annoying problem for quite a few days.
kget is able to resume interrupted file transfers effortlessly.
However wget, curl, axel, lftp when trying to resume, send the appropriate Range Header in HTTP/1.1 but fail to get a reply back. They wait for a long long time, still nothing.
What is it that causes this, and is their a solution, so that I can use wget/curl for the downloads.
Thanks, Nikhil
On Thursday 22 January 2009 23:34, Nikhil Marathe wrote:
Hi all,
I've been having the annoying problem for quite a few days.
kget is able to resume interrupted file transfers effortlessly.
However wget, curl, axel, lftp when trying to resume, send the appropriate Range Header in HTTP/1.1 but fail to get a reply back. They wait for a long long time, still nothing.
What is it that causes this, and is their a solution, so that I can use wget/curl for the downloads.
They work very very well. Are you behind a proxy? In which case you will have to specify the proxy like this http_proxy=http://xxyyzz export $http_proxy
wget -c http://yoururl/file will get your fle and resume on disconnect. You can stop and reissue the same command and wget will resume where it stopped.
On Sat, Jan 24, 2009 at 10:08 AM, jtd jtd@mtnl.net.in wrote:
http_proxy=http://xxyyzz export $http_proxy
There should be $ here.
Regards, NMK.
On Sat, Jan 24, 2009 at 1:50 PM, Nadeem M. Khan nadeem.m.khan@gmail.com wrote:
On Sat, Jan 24, 2009 at 10:08 AM, jtd jtd@mtnl.net.in wrote:
http_proxy=http://xxyyzz export $http_proxy
There should be $ here.
I meant there should be *no* $ here. export http_proxy
Regards, NMK.
On Saturday 24 January 2009 15:21, Nadeem M. Khan wrote:
On Sat, Jan 24, 2009 at 1:50 PM, Nadeem M. Khan
nadeem.m.khan@gmail.com wrote:
On Sat, Jan 24, 2009 at 10:08 AM, jtd jtd@mtnl.net.in wrote:
http_proxy=http://xxyyzz export $http_proxy
There should be $ here.
I meant there should be *no* $ here. export http_proxy
Sorry typo.
Oops, I forgot to mention my connection details.
I'm NOT using a proxy, but I'm connected using Cyberoam client. In linux I'm running linc (linc.sf.net) and set up cron to force it to relogin every 3 minutes.
The problem is not downloading, it is "resuming" of partial downloads. So it is not a question of setting the http_proxy or something. This is some deeper problem
Hope that helps
Thanks Nikhil
On Sat, Jan 24, 2009 at 3:24 PM, jtd jtd@mtnl.net.in wrote:
On Saturday 24 January 2009 15:21, Nadeem M. Khan wrote:
On Sat, Jan 24, 2009 at 1:50 PM, Nadeem M. Khan
nadeem.m.khan@gmail.com wrote:
On Sat, Jan 24, 2009 at 10:08 AM, jtd jtd@mtnl.net.in wrote:
http_proxy=http://xxyyzz export $http_proxy
There should be $ here.
I meant there should be *no* $ here. export http_proxy
Sorry typo.
-- Rgds JTD -- http://mm.glug-bom.org/mailman/listinfo/linuxers
Hi,
On 24-Jan-09, at 22:34, Nikhil Marathe nsm.nikhil@gmail.com wrote:
Oops, I forgot to mention my connection details.
I'm NOT using a proxy, but I'm connected using Cyberoam client. In linux I'm running linc (linc.sf.net) and set up cron to force it to relogin every 3 minutes.
The problem is not downloading, it is "resuming" of partial downloads. So it is not a question of setting the http_proxy or something. This is some deeper problem
Hope that helps
Thanks Nikhil
Yes, the last time I user Linc was a few years back, and I remember that we had to rev engg the cyberoam client protocol to be in keep alive mode, coz they didn't like someone doing this and changed their keep alive logic, this is the precise reason y I need to keep cron n keep relogging on. But this way I think their iptables gets really screwd up and is disconnecting all your active connections, so I think your problem has something definitely to do with linc's keep alive logic.
Since it has been quite long back I do not recollect what I exactly did to fix this problem. If you are good with c n sockets programming, you would be able to submit a fresh hack to linc.
Regards, Mitul Limbani, Founder & CEO, Enterux Solutions, The Enterprise Linux Company(r), http://www.enterux.com/
Hi Nikhil,
Nikhil Marathe wrote:
Oops, I forgot to mention my connection details.
I'm NOT using a proxy, but I'm connected using Cyberoam client. In linux I'm running linc (linc.sf.net) and set up cron to force it to relogin every 3 minutes.
I am not sure about how running linc affects this, but if kget works so should any other client that support resume (ie: most of them).
You mentioned that ...
However wget, curl, axel, lftp when trying to resume, send the appropriate Range Header in HTTP/1.1 but fail to get a reply back. They wait for a long long time, still nothing.
I am assuming that by this you mean: a. You are downloading the files using http (if this is true, is there any specific reason why http instead of ftp ?) b. You used the -c (for wget) or -C (for curl) options (I don't know about the others). c. You are trying these clients for the same server that you use kget for. Since, the ability to resume downloads require server side capability to recognize and accept requests for files from a different offset that the start of the file.
If all of this is true, what is the response you get back from the server for your requests ?
regards, - steve
You mentioned that ...
However wget, curl, axel, lftp when trying to resume, send the
appropriate
Range Header in HTTP/1.1 but fail to get a reply back. They wait for a long long time, still
nothing.
I am assuming that by this you mean: a. You are downloading the files using http (if this is true, is there any specific reason why http instead of ftp ?)
because the download (in this case, google video) only has an http feed. I should point out that cyberoam has blocked FTP for some reason
b. You used the -c (for wget) or -C (for curl) options (I don't know about the others).
I used them
c. You are trying these clients for the same server that you use kget for. Since, the ability to resume downloads require server side capability to recognize and accept requests for files from a different offset that the start of the file.
I tried it on the same files, and got different results.
If all of this is true, what is the response you get back from the server for your requests ?
wget/curl etc. just wait for 5 min or so, then I get bored and kill them.
I'm in college and don't have the luxury of switching providers.
would setting the relogin interval to a higher value help? say 15min.
Thanks
Nikhil
Hi Nikhil,
Nikhil Marathe wrote:
If all of this is true, what is the response you get back from the server for your requests ?
wget/curl etc. just wait for 5 min or so, then I get bored and kill them.
Could you possibly paste the output here (or at pastebin.com, and send us a link ?). You might want to increase verbosity also include the server responses (-S).
If you really want to get to the bottom of this, I'd suggest running tcpdump (normally, i use the options tcpdump -s0 -w <output>.cap host <youripaddress>) and then reading the capture file using wireshark, both with kget and with wget.
would setting the relogin interval to a higher value help? say 15min.
Hmm ...if it isn't required for one client it shouldn't ideally be needed for any other.
regards, - steve
Nikhil Marathe wrote:
Oops, I forgot to mention my connection details.
I'm NOT using a proxy, but I'm connected using Cyberoam client. In linux I'm running linc (linc.sf.net) and set up cron to force it to relogin every 3 minutes.
The problem is not downloading, it is "resuming" of partial downloads. So it is not a question of setting the http_proxy or something. This is some deeper problem
It is your ISP's service that collapses on you. You can't do much except try out another internet provider if available.
-
jtd -
Mitul Limbani -
Nadeem M. Khan -
Nikhil Marathe -
Rony -
steve