暗无天日

=============>DarkSun的个人博客

curl练习

从 https://jvns.ca/blog/2019/08/27/curl-exercises/?utm_source=programmingdigest&utm_medium=web&utm_campaign=featured 上看到了一些 curl 的练习，觉得不错，尝试做一下，增强对 curl 的理解。

curl 的用法

curl.jpg

练习

  • Request https://httpbin.org 

    curl https://httpbin.org |head

    <!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <title>httpbin.org</title>
    <link href="https://fonts.googleapis.com/css?family=Open+Sans:400,700|Source+Code+Pro:300,600|Titillium+Web:400,600,700"
        rel="stylesheet">
    <link rel="stylesheet" type="text/css" href="/flasgger_static/swagger-ui.css">
    <link rel="icon" type="image/png" href="/static/favicon.ico" sizes="64x64 32x32 16x16" />

  • Request https://httpbin.org/anything. httpbin.org/anything will look at the request you made, parse it, and echo back to you what you requested. curl’s default is to make a GET request. 

    curl https://httpbin.org/anything

    {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0"
  }, 
  "json": null, 
  "method": "GET", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything"
}

  • Make a POST request to https://httpbin.org/anything 

    curl -X post https://httpbin.org/anything

    {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0"
  }, 
  "json": null, 
  "method": "POST", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything"
}

  • Make a GET request to https://httpbin.org/anything, but this time add some query parameters (set value=panda). 

    curl 'https://httpbin.org/anything?name=darksun&id=1'

    {
  "args": {
    "id": "1", 
    "name": "darksun"
  }, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0"
  }, 
  "json": null, 
  "method": "GET", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything?name=darksun&id=1"
}

  • Request google’s robots.txt file (www.google.com/robots.txt) 

    https_proxy=192.168.1.9:8118 curl https://www.google.com/robots.txt

    User-agent: *
Disallow: /search
Allow: /search/about
Allow: /search/static
Allow: /search/howsearchworks
Disallow: /sdch
Disallow: /groups
Disallow: /index.html?
Disallow: /?
Allow: /?hl=
Disallow: /?hl=*&
Allow: /?hl=*&gws_rd=ssl$
Disallow: /?hl=*&*&gws_rd=ssl
Allow: /?gws_rd=ssl$
Allow: /?pt1=true$
Disallow: /imgres
Disallow: /u/
Disallow: /preferences
Disallow: /setprefs
Disallow: /default
Disallow: /m?
Disallow: /m/
Allow:    /m/finance
Disallow: /wml?
Disallow: /wml/?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/?
Disallow: /pda/search?
Disallow: /sprint_xhtml
Disallow: /sprint_wml
Disallow: /pqa
Disallow: /palm
Disallow: /gwt/
Disallow: /purchases
Disallow: /local?
Disallow: /local_url
Disallow: /shihui?
Disallow: /shihui/
Disallow: /products?
Disallow: /product_
Disallow: /products_
Disallow: /products;
Disallow: /print
Disallow: /books/
Disallow: /bkshp?*q=*
Disallow: /books?*q=*
Disallow: /books?*output=*
Disallow: /books?*pg=*
Disallow: /books?*jtp=*
Disallow: /books?*jscmd=*
Disallow: /books?*buy=*
Disallow: /books?*zoom=*
Allow: /books?*q=related:*
Allow: /books?*q=editions:*
Allow: /books?*q=subject:*
Allow: /books/about
Allow: /booksrightsholders
Allow: /books?*zoom=1*
Allow: /books?*zoom=5*
Allow: /books/content?*zoom=1*
Allow: /books/content?*zoom=5*
Disallow: /ebooks/
Disallow: /ebooks?*q=*
Disallow: /ebooks?*output=*
Disallow: /ebooks?*pg=*
Disallow: /ebooks?*jscmd=*
Disallow: /ebooks?*buy=*
Disallow: /ebooks?*zoom=*
Allow: /ebooks?*q=related:*
Allow: /ebooks?*q=editions:*
Allow: /ebooks?*q=subject:*
Allow: /ebooks?*zoom=1*
Allow: /ebooks?*zoom=5*
Disallow: /patents?
Disallow: /patents/download/
Disallow: /patents/pdf/
Disallow: /patents/related/
Disallow: /scholar
Disallow: /citations?
Allow: /citations?user=
Disallow: /citations?*cstart=
Allow: /citations?view_op=new_profile
Allow: /citations?view_op=top_venues
Allow: /scholar_share
Disallow: /s?
Allow: /maps?*output=classic*
Allow: /maps?*file=
Allow: /maps/api/js?
Allow: /maps/d/
Disallow: /maps?
Disallow: /mapstt?
Disallow: /mapslt?
Disallow: /maps/stk/
Disallow: /maps/br?
Disallow: /mapabcpoi?
Disallow: /maphp?
Disallow: /mapprint?
Disallow: /maps/api/js/
Disallow: /maps/api/place/js/
Disallow: /maps/api/staticmap?
Disallow: /maps/api/streetview?
Disallow: /maps/api/streetview/
Disallow: /maps/_/sw/manifest.json
Disallow: /mld?
Disallow: /staticmap?
Disallow: /maps/preview
Disallow: /maps/place
Disallow: /maps/timeline/
Disallow: /help/maps/streetview/partners/welcome/
Disallow: /help/maps/indoormaps/partners/
Disallow: /lochp?
Disallow: /center
Disallow: /ie?
Disallow: /blogsearch/
Disallow: /blogsearch_feeds
Disallow: /advanced_blog_search
Disallow: /uds/
Disallow: /chart?
Disallow: /transit?
Allow:    /calendar$
Allow:    /calendar/about/
Disallow: /calendar/
Disallow: /cl2/feeds/
Disallow: /cl2/ical/
Disallow: /coop/directory
Disallow: /coop/manage
Disallow: /trends?
Disallow: /trends/music?
Disallow: /trends/hottrends?
Disallow: /trends/viz?
Disallow: /trends/embed.js?
Disallow: /trends/fetchComponent?
Disallow: /trends/beta
Disallow: /trends/topics
Disallow: /musica
Disallow: /musicad
Disallow: /musicas
Disallow: /musicl
Disallow: /musics
Disallow: /musicsearch
Disallow: /musicsp
Disallow: /musiclp
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /wapsearch?
Allow: /safebrowsing/diagnostic
Allow: /safebrowsing/report_badware/
Allow: /safebrowsing/report_error/
Allow: /safebrowsing/report_phish/
Disallow: /reviews/search?
Disallow: /orkut/albums
Disallow: /cbk
Allow: /cbk?output=tile&cb_client=maps_sv
Disallow: /maps/api/js/AuthenticationService.Authenticate
Disallow: /maps/api/js/QuotaService.RecordEvent
Disallow: /recharge/dashboard/car
Disallow: /recharge/dashboard/static/
Disallow: /profiles/me
Allow: /profiles
Disallow: /s2/profiles/me
Allow: /s2/profiles
Allow: /s2/oz
Allow: /s2/photos
Allow: /s2/search/social
Allow: /s2/static
Disallow: /s2
Disallow: /transconsole/portal/
Disallow: /gcc/
Disallow: /aclk
Disallow: /cse?
Disallow: /cse/home
Disallow: /cse/panel
Disallow: /cse/manage
Disallow: /tbproxy/
Disallow: /imesync/
Disallow: /shenghuo/search?
Disallow: /support/forum/search?
Disallow: /reviews/polls/
Disallow: /hosted/images/
Disallow: /ppob/?
Disallow: /ppob?
Disallow: /accounts/ClientLogin
Disallow: /accounts/ClientAuth
Disallow: /accounts/o8
Allow: /accounts/o8/id
Disallow: /topicsearch?q=
Disallow: /xfx7/
Disallow: /squared/api
Disallow: /squared/search
Disallow: /squared/table
Disallow: /qnasearch?
Disallow: /app/updates
Disallow: /sidewiki/entry/
Disallow: /quality_form?
Disallow: /labs/popgadget/search
Disallow: /buzz/post
Disallow: /compressiontest/
Disallow: /analytics/feeds/
Disallow: /analytics/partners/comments/
Disallow: /analytics/portal/
Disallow: /analytics/uploads/
Allow: /alerts/manage
Allow: /alerts/remove
Disallow: /alerts/
Allow: /alerts/$
Disallow: /ads/search?
Disallow: /ads/plan/action_plan?
Disallow: /ads/plan/api/
Disallow: /ads/hotels/partners
Disallow: /phone/compare/?
Disallow: /travel/clk
Disallow: /travel/hotelier/terms/
Disallow: /hotelfinder/rpc
Disallow: /hotels/rpc
Disallow: /commercesearch/services/
Disallow: /evaluation/
Disallow: /chrome/browser/mobile/tour
Disallow: /compare/*/apply*
Disallow: /forms/perks/
Disallow: /shopping/suppliers/search
Disallow: /ct/
Disallow: /edu/cs4hs/
Disallow: /trustedstores/s/
Disallow: /trustedstores/tm2
Disallow: /trustedstores/verify
Disallow: /adwords/proposal
Disallow: /shopping/product/
Disallow: /shopping/seller
Disallow: /shopping/ratings/account/metrics
Disallow: /shopping/reviewer
Disallow: /about/careers/applications/
Disallow: /landing/signout.html
Disallow: /webmasters/sitemaps/ping?
Disallow: /ping?
Disallow: /gallery/
Disallow: /landing/now/ontap/
Allow: /searchhistory/
Allow: /maps/reserve
Allow: /maps/reserve/partners
Disallow: /maps/reserve/api/
Disallow: /maps/reserve/search
Disallow: /maps/reserve/bookings
Disallow: /maps/reserve/settings
Disallow: /maps/reserve/manage
Disallow: /maps/reserve/payment
Disallow: /maps/reserve/receipt
Disallow: /maps/reserve/sellersignup
Disallow: /maps/reserve/payments
Disallow: /maps/reserve/feedback
Disallow: /maps/reserve/terms
Disallow: /maps/reserve/m/
Disallow: /maps/reserve/b/
Disallow: /maps/reserve/partner-dashboard
Disallow: /about/views/
Disallow: /intl/*/about/views/
Disallow: /local/dining/
Disallow: /local/place/products/
Disallow: /local/place/reviews/
Disallow: /local/place/rap/
Disallow: /local/tab/
Allow: /finance
Allow: /js/
Disallow: /finance?*q=*

# AdsBot
User-agent: AdsBot-Google
Allow: /maps/api/js?
Disallow: /maps/api/js/
Disallow: /maps/api/place/js/
Disallow: /maps/api/staticmap?
Disallow: /maps/api/streetview?
Disallow: /maps/api/streetview/

# Certain social media sites are whitelisted to allow crawlers to access page markup when links to google.com/imgres* are shared. To learn more, please contact images-robots-whitelist@google.com.
User-agent: Twitterbot
Allow: /imgres

User-agent: facebookexternalhit
Allow: /imgres

Sitemap: https://www.google.com/sitemap.xml

  • Make a GET request to https://httpbin.org/anything and set the header User-Agent: elephant. 

    curl -H "User-Agent: elephant" 'https://httpbin.org/anything'

    {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "elephant"
  }, 
  "json": null, 
  "method": "GET", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything"
}

  • Make a DELETE request to https://httpbin.org/anything 

    curl -X delete 'https://httpbin.org/anything'

    {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0"
  }, 
  "json": null, 
  "method": "DELETE", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything"
}

  • Request https://httpbin.org/anything and also get the response headers 

    curl -i 'https://httpbin.org/anything'

    HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 11 Sep 2019 07:43:57 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 286
Connection: keep-alive

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0"
  }, 
  "json": null, 
  "method": "GET", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything"
}

  • Make a POST request to https://httpbin.com/anything with the JSON body {"value": "panda"} 

    curl -X post 'https://httpbin.org/anything' --data '{"value":"panda"}'

    {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "{\"value\":\"panda\"}": ""
  }, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "17", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0"
  }, 
  "json": null, 
  "method": "POST", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything"
}

  • Make the same POST request as the previous exercise, but set the Content-Type header to application/json (because POST requests need to have a content type that matches their body). Look at the json field in the response to see the difference from the previous one. 

    curl -X post -H 'Content-Type: application/json' 'https://httpbin.org/anything' --data '{"value":"panda"}'

    {
  "args": {}, 
  "data": "{\"value\":\"panda\"}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "17", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0"
  }, 
  "json": {
    "value": "panda"
  }, 
  "method": "POST", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything"
}

  • Make a GET request to https://httpbin.org/anything and set the header Accept-Encoding: gzip (what happens? why?) 

    curl -X post -H 'Accept-Encoding: gzip' 'https://httpbin.org/anything' |gzip -d

    {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0"
  }, 
  "json": null, 
  "method": "POST", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything"
}

  • Put a bunch of a JSON in a file and then make a POST request to https://httpbin.org/anything with the JSON in that file as the body 

    tmp=$(mktemp)
title="TEST OF JQ"
source="JQ NB"
echo '{}'|jq '{"title":$title,"content":$source}' --arg title "$title" --arg source "$source" >${tmp}
curl -X post -H 'Content-Type: application/json'  'https://httpbin.org/anything' --data @${tmp}

    {
  "args": {}, 
  "data": "{  \"title\": \"TEST OF JQ\",  \"content\": \"JQ NB\"}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "46", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0"
  }, 
  "json": {
    "content": "JQ NB", 
    "title": "TEST OF JQ"
  }, 
  "method": "POST", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything"
}

  • Make a request to https://httpbin.org/image and set the header ‘Accept: image/png’. Save the output to a PNG file and open the file in an image viewer. Try the same thing with with different Accept: headers. 

    png="images/curl_execise_1.png"
curl -H "Accept: image/png" https://httpbin.org/image >${png}
printf ${png}

    curl_execise_1.png

  • Make a PUT request to https://httpbin.org/anything 

    curl -X put  https://httpbin.org/anything

    {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0"
  }, 
  "json": null, 
  "method": "PUT", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything"
}

  • Request https://httpbin.org/image/jpeg, save it to a file, and open that file in your image editor. 

    img="images/curl_execise_2.jpg"
curl https://httpbin.org/image/jpeg >${img}
printf ${img}

    curl_execise_2.jpg

  • Request https://www.twitter.com. You’ll get an empty response. Get curl to show you the response headers too, and try to figure out why the response was empty. 

    https_proxy=192.168.1.9:8118 curl -i https://www.twitter.com

    HTTP/1.1 200 Connection established

HTTP/2 301 
content-length: 0
date: Thu, 12 Sep 2019 12:51:19 GMT
location: https://twitter.com/

server: tsa_m
set-cookie: personalization_id="v1_qj2JqquTZhCMoXb2fis0og=="; Max-Age=63072000; Expires=Sat, 11 Sep 2021 12:51:19 GMT; Path=/; Domain=.twitter.com
set-cookie: guest_id=v1%3A156829267991786375; Max-Age=63072000; Expires=Sat, 11 Sep 2021 12:51:19 GMT; Path=/; Domain=.twitter.com
strict-transport-security: max-age=631138519
x-connection-hash: c6824d2630b6c4b240a7683f2f8e08d2
x-response-time: 100

  • Make any request to https://httpbin.org/anything and just set some nonsense headers (like panda: elephant) 

    curl -H "panda: elephant" https://httpbin.org/anything

    {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "Panda": "elephant", 
    "User-Agent": "curl/7.58.0"
  }, 
  "json": null, 
  "method": "GET", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything"
}

  • Request https://httpbin.org/status/404 and https://httpbin.org/status/200. Request them again and get curl to show the response headers. 

    curl -i  https://httpbin.org/status/404
curl -i  https://httpbin.org/status/200

    HTTP/1.1 404 NOT FOUND
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: text/html; charset=utf-8
Date: Wed, 11 Sep 2019 11:20:41 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 0
Connection: keep-alive

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: text/html; charset=utf-8
Date: Wed, 11 Sep 2019 11:20:43 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 0
Connection: keep-alive

  • Request https://httpbin.org/anything and set a username and password (with -u username:password) 

    curl -u lujun9972:1233 https://httpbin.org/anything

    {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Authorization": "Basic bHVqdW45OTcyOjEyMzM=", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0"
  }, 
  "json": null, 
  "method": "GET", 
  "origin": "61.142.7.18, 61.142.7.18", 
  "url": "https://httpbin.org/anything"
}

  • Download the Twitter homepage (https://twitter.com) in Spanish by setting the Accept-Language: es-ES header. 

    https_proxy=192.168.1.9:8118 curl -H "Accept-Language: es-ES" https://twitter.com |tail

    
    <input type="hidden" class="swift-boot-module" value="app/pages/static_lohp">
  <input type="hidden" id="swift-module-path" value="https://abs.twimg.com/k/swift/es">


    <script src="https://abs.twimg.com/k/es/init.es.0df6a2c05184e170f1b7.js" async></script>

  </body>
</html>
2019-09-11 2019-09-12 linux和它的小伙伴 lujun9972

Generated by Emacs 27.x(Org mode 9.x)

Copyright © 2014 - lujun9972   -   Powered by EGO
Themed with emacs_love Creative Commons License