Tangerine v4.0.0 Planning

[chrisekelley]

Since we're making so many upgrades to the libraries that Tangerine depends upon, let's make the journey to 4.0. Here are the results of a planning discussion w/ Erik this morning:

• Name this release v4.0.0
• Documentation about the v4.0.0 release, including what we learned in the v3.x line (case studies, how many deployments, load, etc).
• Shift to new in-browser database using cryptopouch away from sqlite db.
• Improve user interface in Editor for managing case documents featuring more fine-grained deletions, archiving, and conflict resolution. Migrate Editor's case document listing to a more tabular format featuring created/modified dates and relevant action buttons for each record.
• Enable deletion from tablet of old cases using the new Pouchdb deletion code
• Document process for installing ClamAV on servers.
• Feature: Expand new group configuration (i.e. Wizard) [Feature: Expand new group configuration (i.e. Wizard) #3481](Feature: Expand new group configuration (i.e. Wizard) #3481 - this can be expanded!
• Fix saving of device logs once-and-for-all. (We're still using the wedge script, aren't we?)
• Automate device registration for projects that don't need such fine-grained tracking of devices.
• Remove support of sync-protocol-1.
• Document Tangerine Data API's.

[esurface]

For Discussion

These are copied from the 2022 brainstorm ticket.

Data Management Story

Allow the data manager to manipulate all types of data with the general rule: If you can do something, you can undo it as well.

1. Data Manager can review all forms linked to a case (whether they are in the case list or not) and choose which forms to link to the case
2. Data Manager can archive or delete individual form responses
3. Data Manager can archive or delete participants
4. Data Manager can archive or delete case events
5. Cases that are Closed don't sync down -- (added from thoughts on stand-up on Jan 27, 2022

Tablet Storage Maintenance Story

New tools in the 'Maintenance' section of the app

1. Tablet Administrator can compress form revisions to reduce storage usage (may be available in v3.21 or v3.22)
2. Tablet Admin can delete incomplete unlinked forms (may be able to handle with on-update scripts)

Platform Improvements

1. Update to latest pouchdb 8.0 database (check sync tracking)

Conflict Management

1. Auto-conflict resolution

[chrisekelley]

Thinking about one of the most difficult parts of our core mission - making sure that data collected on tablets is uploaded efficiently and completely. Is there room in our current budget to improve this part? For example, I am syncing a tablet that will have over 100k documents. Is the messaging correct and translated? I think the number used for total docs is incorrect - it is actually the count of changes "docs" not cases or formResponses. Maybe that needs to be a percentage. There also seems to be long pauses where there is no message, just the lines. What is happening? Can we tell the user what is happening in every step that take more than a couple seconds?

[chrisekelley]

Another observation of a device that has almost 20k docs. Rendering the Case home listing is a tad slow. Not terrible, but I notice it is slow. Can we improve this? Are there changes in pouchdb 8.0 that could help us?

[esurface]

Future Improvements

Things we'd love do:

• Update everything using Polymer/Paper to either Lit or skip to Material Design (Start with client updater app)
• Move for definitions from HTML to JSON
• Move form logic javascript to javascript editor
• Get Tangerine Preview with Case running in tangerine editor
• Explore changing Authentication code to no longer rely on CouchDB
• Group and App Configuration GUI
• GitHub integration

[TSSlade]

What is the benefit of moving to JSON from HTML for form definitions? What do we lose along the way?

If forms were articulated as JSON, would we still have the ability to apply rich formatting to the item prompts, item labels, etc.?

[Michael833]

Yes, I have questions about this also. I believe, from informal discussions with others who program instruments regularly, that most people use the web interface to create. I use HTML and code directly. Will that still be possible?

[TSSlade]

@Michael833 - I am 99% certain this is a question about the back-end representation of the instruments. I don't believe there's any discussion of removing the "wizard"/ front-end UI that non-devs currently use for building tools in Tangerine.

Nevermind, just realized that I had misread your question. But leaving this here in case relevant for others.

You can still write JSON directly into a text editor (as you would for HTML). It would definitely take a bit of reshaping your code snippets, and I suspect it would make things substantially less ergonomic in the short-term.

[Michael833]

Or the instrument's HTML important function?

…

-Michael -- Michael Costello "To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination: he may be able to say what the experiment died of." -Sir Ronald Aylmer Fisher, FRS "Here are some additional !!!! if I didn’t use enough. Please put them wherever makes you feel appreciated and like I’m friendly and fun but still cool and professional. Thanks.” -James Hamblin Tiny keyboard, so please excuse typos, brevity, and forgetting what I was saying mid-....

On Fri, Apr 28, 2023, 1:13 PM T.Slade ***@***.***> wrote: @Michael833 <https://github.com/Michael833> - I am 99% certain this is a question about the back-end representation of the instruments. I don't believe there's any discussion of removing the "wizard"/ front-end UI that non-devs currently use for building tools in Tangerine. — Reply to this email directly, view it on GitHub <#3507 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFZQXOZWE2SBFA2RLRIRNDXDPUDXANCNFSM6AAAAAAT6JUN3Y> . You are receiving this because you were mentioned.Message ID: <Tangerine-Community/Tangerine/repo-discussions/3507/comments/5756400@ github.com>

[TSSlade]

Not sure I understand -- too brief! Care to elaborate?

[Michael833]

It feels like the words "back-end" (and therefore "front-end") probably have different meanings for different people. Let me define how I think of it: Front-end: The Tangerine website UI. I can create instruments here, download data, edit instruments, etc. Back-end: The actual HTML that the INSTRUMENT is built off of. With all the extra spaces that the runner puts in when validating and error checking. The-real-back-end: The runner, the code used to create the front-end website, the JSON files, etc. I want to ensure that the "back-end" door is still there, and that I can create and edit instruments using something else (Notepad++, Sublime, etc.) and then paste them into the text box on the "front-end" in order for "the-real-back-end" to render them into .apk files or PWAs.

…

On Fri, Apr 28, 2023 at 1:56 PM T.Slade ***@***.***> wrote: Not sure I understand -- too brief! Care to elaborate? — Reply to this email directly, view it on GitHub <#3507 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFZQXIG6EBUUATRIJOTG5LXDQAEZANCNFSM6AAAAAAT6JUN3Y> . You are receiving this because you were mentioned.Message ID: <Tangerine-Community/Tangerine/repo-discussions/3507/comments/5757262@ github.com>

[TSSlade]

Sounds like you're wanting to make sure there's still a mechanism for creating forms distinct from the Tangerine Editor User Interface. I think retaining that is non-controversial, and also implicit in Erik's note about moving form definitions to JSON from HTML. But good to have that explicitly recorded here.