2021-10-11 Development Meeting

Kacper, Craig, Mike H, Mike L, Tim


  • Updates
  • Remote bags – possible implementations
    • Deriva will deposit their data/metadata as bags in S3 (or S3-like buckets)
    • Deposits will have uuids that could be translated to particular urls
    • How do we treat that?
      • MH: depositing to s3 may be asynchronous
      • Options:
        1. Fetch the zip, unpack, upload raw bytes to tale DataDir
        2. Register zip url, cache in DMS, register files as arcp://<zip_id>/path_inside_zip
        3. arcp://<deriva|s3|uuid>/path_inside_zip, cache individual files in dms that were accessed via range request


  • Kacper
  • Mike L
    • No updates this week.
  • Tim
    • Recording of processes with bpftrace now fully under control.
      • Lots of stuff to filter out to make the info manageable.
      • Surprised by number of failed execs made by nbconvert.
      • Pretty horrified by the effect of using LD_LIBRARY_PATH.
    • Now focusing on I/O.
      • A bit more manageable now with focus on relevant I/O.
  • Craig
  • Mike H