2019-09-09: Development Meeting

Kacper, Tommy, Elias, Craig, Joe, Mike L, Mike H

Agenda

  • Updates
  • Discussion
    • Planning ([name=Craig])
      • In progress
        • Angular migration
        • Publish to Zenodo (backend)
        • Tapis/bdbag integration
        • Kubernetes Helm chart
        • External Oauth integration
      • UI design needed
        • User settings UI (needed by publish to Zenodo)
        • Create tale from Git repo
        • Sharing (PI team priority)
      • Design needed
        • Tales at scale
          • What is the middleware and API we need to use to submit jobs?
          • Can we decompose “gwvolman” into different operations with authentication
          • Will have UI work
        • Provenance capture (“reproducible run”)
          • Possible tasks:
            • Enable ptrace cap on containers
            • UI for environment config
            • Auto-detecting changes to the environment (prototype)
              • Compare differences between running container and base image
              • Sciunit/Reprozip: detecting what was actually used during execution
        • Stata / Matlab support (supplement)
        • Publishing refactor (deferred)
          • Girder plugin
          • Supporting drafts and curation workflows
        • Supporting verification workflows
    • Questions about Files API ([name=Mike L])
      • Do we want Run > Files to be able to navigate into Datasets? If yes, do we want to prohibit deleting folders/items within a Dataset?
        • No. We should prevent this.
      • Can we Remove/Delete entire folders (from Home/Tale Workspace)? Must they be empty to remove them? Yes, and they don’t need to be empty.
        • I think this may have been a regression from this
      • How do we know when an upload is complete? Is there some signal that I’m missing or some way to check that a given uploadId is complete?
        • Minimum is apparently 5 MB
      • Tommy Note: At one point we had the ember drop-zone widget where users could drag and drop files and folders. Somewhere along the line this broke (before the big user meetup) and was requested (I can find the issue). Not sure if this is in scope or too late
        • See https://github.com/whole-tale/dashboard/issues/167
        • See https://github.com/whole-tale/whole-tale/issues/39

Updates

  • Kacper
    • Finishing Zenodo publish integration (PR incoming)
    • Tagged and deployed v0.8rc1 (https://dashboard.stage.wholetale.org)
      • girderfs
        • Better error handling (girderfs#12)
      • gwvolman
        • Improvements to export to DataONE (gwvolman#72,74,84)
        • Tale build logs are cleaner (gwvolman#80)
        • Fix Analyze in WT for generic HTTP resources (gwvolman#82)
      • wt_data_manager
        • Better error handling for unreachable resources (wt_data_manager#42)
      • wholetale
        • Allow to import Tales from zipfiles (wholetale#259)
        • Allow to import “binder-like” external datasets as Tales (wholetale#317)
        • Support hierarchy in Dataverse packages (wholetale#318)
        • Fix Globus integration to use new petrel.net (wholetale#339)
        • Fix citation generation for http and Globus resources (wholetale#328,315)
        • Improvements to the notification system (wholetale#305)
        • Improvements and bugfixes for exporting Tales (wholetale#310,321,345,342)
      • dashboard
        • “Copy on Launch” (dashboard#448,459,478)
        • Improvements to the notification system (dashboard#461,482,491)
        • “Browse”-tab refactor
    • Documentation PR reviews
    • ADC publishing discussion
  • Tommy
    • Meeting with Julient Brun about LTER using Whole Tale for reproducability
      • He’s having issues with scientists submitting code+data and not being able to re-run it
        • Having issues where it works on the researchers’ machines but not his
          • He’s hopeful that something like Whole Tale can cut down on those instances because it’s a fresh environment every time
        • He’s also interested in getting an automated capture of which files were used which scripts
        • But it sounds like he’s more interested in higher level metrics such as the execution order and required packages
        • Comment: He really likes the feature where you can download the Tale and run/share it locally
    • Update from last meeting on storing file paths in DataONE
      • Agreement across the team on using prov:atLocation in the resource map
      • Requires adding/refactoring d1_python and the R client to add paths
      • Also requires fiddling with the export format.
        • The link posted in the last meeting has more details on this
        • tldr: This will be a second version of their export format
        • Plan is to first deal with clients (python, R)
        • Add the functionality to Whole Tale
        • Fiddle with Metacat and GMN (DataONE members can choose if they want to deploy the member node server in Java or in Python)
    • Currently working on the bug where Tales have the Run button when coming from a redirect from the New Tale Modal
      • It’s looking like the instance hasn’t been defined on the Tale model yet (it’s undefined by the time the handlebars template is parsed)
    • Met with Jeanette Clark from ADC with Kacper and Craig
      • We’ve disabled publishing to the ADC until we have integration with a curation workflow
      • To keep things general, this will probably look like a “draft” publication to repository X
        • There are many different repositories out there that require different metadata fields. Instead of creating a UI tool that replicates all of them, it makes more sense to. publish a draft and have the user use their editor
        • Possibly ask for information about the state of the package on Tale launch
  • Elias
    • Finished Tapis app tutorial
    • Started learning bdbag cli
    • Discovered and reported a bug in bdbag Github
    • Started working on bdbag Tapis task including design document. Task
  • Craig
    • Gateways poster/ParCo paper
    • RO paper submitted (previous week) – will be at Gateways/eScience
    • PR review for v0.8
    • Planning
    • UX meeting this week (Zenodo/API keys)
  • Mike L
    • Angular refactoring: Run > Files view
      • Added Create Folder / Upload File
        • File uploads are confusing since we need to send file bytes as body - parameters not captured by Girder’s incomplete Swagger spec
        • Need to read existing UI code to find which endpoints to hit in the correct order to upload a file without error
      • Added dropdown options to each file/folder
        • Download file/folder
        • Copy file/folder
        • Delete file/folder
        • Rename file/folder
        • Move file/folder (untested - needs modal)
    • I am trying to keep note of all of the problems with Girder’s swagger spec, but the problems are numerous
      • Missing parameters (e.g. chunk or file contents in body)
      • Missing models/return types (e.g. Response Class: “Item is not defined!”)
        • Upload
        • Item
        • etc
    • A few low-level questions regarding Files API (see “Discussion” above)
  • Mike H
    • Finished ParCo paper