Connect and share knowledge within a single location that is structured and easy to search. I have a strange problem with Curl on Ubuntu Each night i Curl a large file from a remote API endpoint and save it to a folder, i use the curl command inside a bash script, i have noticed that my Curl command fails to save the output most of the time, so i created a script that checks if the file has saved, and if not it attempts to download again. Each night i check the logs and i can see see that most of the time the curl fails at-least 5 times before saving the file:.
The strange thing is when i run curl without the apostrophes round the URL and don't save it to a file:. I see the file downloads, but the command output does not go into file. If i use -o with Curl to specify the output i get the same timeout issue as above and it often fails multiple times before working. Can anybody shed light on this and show a proper method of curling a large file? I think the server takes too much time to retrieve the file or something alike.
Once it's requested several times, the server loads it somewhere for 'quick access', which is why not every command fails.
You could try to add the --connect-timeout option to give the server more time to response 5 minutes I would suggest. If this option is used several times, the last one will be used.
If the file indeed is too large, you could go one and add the -m option to extend the time you're allowed to download time depends on your connection speed.
This is useful for preventing your batch jobs from hanging for hours due to slow networks or links going down. See also the --connect-timeout option. If everything still goes wrong, try to add the -D option and change the -s option to -v. This option is handy to use when you want to store the headers that a HTTP site sends to you. Mostly useful for debugging. However, if I try to download the same URL from the command line using curl , it doesn't work—the result is an empty file:.
Of course I can get the file through the browser, but I would like to understand why the curl command above doesn't work. Why can't curl download this file correctly, when it is evidently present on the website and can be correctly accessed and downloaded through a graphical web browser?
Because it's a CDN, the exact behaviour whether you get redirected or not might depend on your location. To tell it to do so, add the -L argument:. If the browser is able to download the file, you can inspect what the browser is doing. Now you have a curl link that will work.
It will probably have excess parameters you can trim-away. So this will result in a working response from a web browser but not a working response in a backend web request like curl. I just ran into a site that required all the following headers. Failure to specify them resulted in a timeout. Sign up to join this community. The best answers are voted up and rise to the top.
Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Learn more. Why won't curl download this link when a browser will? So what if we want to use cURL to save a file to our server? For that, we must use the -O option:. You'll notice that cURL will display a download progress table rather than the file contents this time:. If you'd like the file to have a different file name perhaps readme is too bossy for you , specify it after -O :.
That's all well and good, but downloading lots of files in this way would quickly become a hassle. You can download more than one file in a single command by using the following syntax:.
If you have a long list of different files you want to download, you can place them in a text file and run cURL with xargs:. As it's common for site admins to move the location of a file, then redirect to the new one, it can be good practice to include the -L option in your cURL command.
0コメント