2019-03-08: Development Planning

Kacper, Craig

Discussion of HTTP registration:

  • Looking at examples:
    • From https://github.com/bocinsky/guedesbocinsky2018
      • ftp://ftp.ncdc.noaa.gov/pub/data/paleo/contributions_by_author/mann2008/instrument.zip
    • Liri’s tale
      • https://www2.census.gov/programs-surveys/ahs/2011/AHS%202011%20National%20and%20Metropolitan%20PUF%20v1.4%20CSV.zip
    • LIGO tale
      • https://doi.org/10.7935/K53X84K2
      • https://www.gw-openscience.org/GW170104data/H-H1_LOSC_4_V1-1167559920-32.hdf5
    • Illustris example
      • http://use.yt/upload/9241131f
      • illustris.jpg
      • use.yt/upload/9241131f/illustris.jpg
    • XKAN example
      • https://data.illinois.gov/dataset/68ac778a-2e06-47c1-ba22-bc608fe1fd85/resource/69c8e2a9-c7cb-46af-bfed-4cdd7f213e2d/download/bid_data_on_projects_for_the_past_10_years.csv
      • read(“/data/data.illinois.gov/dataset/68ac778a-2e06-47c1-ba22-bc608fe1fd85/resource/69c8e2a9-c7cb-46af-bfed-4cdd7f213e2d/download/bid_data_on_projects_for_the_past_10_years.csv”)
      • get(“http://data.illinois.gov/dataset/68ac778a-2e06-47c1-ba22-bc608fe1fd85/resource/69c8e2a9-c7cb-46af-bfed-4cdd7f213e2d/download/bid_data_on_projects_for_the_past_10_years.csv”)
    • Given no alternative, the best information for registration and caching is the complete URL including path hierarchy
      • Protocol is some level of provider
      • ftp -> FTP
      • http https -> HTTP
        • Includes DataONE/Dataverse
      • globus: -> Globus
      • agave: -> Agave
wifi-60-214:~ willis8$ curl -I http://use.yt/upload/9241131f
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 08 Mar 2019 20:36:29 GMT
Content-Type: application/octet-stream
Content-Length: 781665
Connection: keep-alive
Content-Disposition: attachment; filename=illustris.jpg
Etag: "da39a3ee5e6b4b0d3255bfef95601890afd80709"
Strict-Transport-Security: max-age=31536000
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Clacks-Overhead: GNU Terry Pratchett
Referrer-Policy: strict-origin-when-cross-origin

wifi-60-214:~ willis8$ curl -I https://www2.census.gov/programs-surveys/ahs/2011/AHS%202011%20National%20and%20Metropolitan%20PUF%20v1.4%20CSV.zip
HTTP/1.1 200 OK
Server: Apache
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Cache-Control: private
Last-Modified: Tue, 03 Feb 2015 13:01:45 GMT
ETag: "46a0a3b-c1cfb48-50e2eac446040"
Accept-Ranges: bytes
Content-Length: 203225928
Content-Type: application/zip
Strict-Transport-Security: max-age=31536000
Date: Fri, 08 Mar 2019 20:36:42 GMT
Connection: keep-alive

wifi-60-214:~ willis8$ curl -I http://data.illinois.gov/dataset/68ac778a-2e06-47c1-ba22-bc608fe1fd85/resource/69c8e2a9-c7cb-46af-bfed-4cdd7f213e2d/download/bid_data_on_projects_for_the_past_10_years.csv
HTTP/1.1 301 Moved Permanently
Server: nginx/1.13.9
Date: Fri, 08 Mar 2019 20:37:05 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://data.illinois.gov/dataset/68ac778a-2e06-47c1-ba22-bc608fe1fd85/resource/69c8e2a9-c7cb-46af-bfed-4cdd7f213e2d/download/bid_data_on_projects_for_the_past_10_years.csv

wifi-60-214:~ willis8$ curl -I https://data.illinois.gov/dataset/68ac778a-2e06-47c1-ba22-bc608fe1fd85/resource/69c8e2a9-c7cb-46af-bfed-4cdd7f213e2d/download/bid_data_on_projects_for_the_past_10_years.csv
HTTP/1.1 200 OK
Server: nginx/1.13.9
Date: Fri, 08 Mar 2019 20:37:12 GMT
Content-Type: text/csv
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Accept-Ranges: bytes
Last-Modified: Wed, 12 Dec 2018 18:25:28 GMT
Content-Range: bytes 0-1155455/1155456
ETag: "1544639128.19-1155456"