Merge pull request #6369 from IQSS/develop

v4.18

Author: kcondon

README.md

@@ -17,7 +17,9 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 
 [![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg?raw=true "Dataverse Project")](http://dataverse.org)
 
-[![Build Status](https://travis-ci.org/IQSS/dataverse.svg?branch=develop)](https://travis-ci.org/IQSS/dataverse) [![Coverage Status](https://coveralls.io/repos/IQSS/dataverse/badge.svg?branch=develop&service=github)](https://coveralls.io/github/IQSS/dataverse?branch=develop)
+[![API Test Status](https://jenkins.dataverse.org/buildStatus/icon?job=IQSS-dataverse-develop&subject=API%20Test%20Status)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/)
+[![Unit Test Status](https://img.shields.io/travis/IQSS/dataverse?label=Unit%20Test%20Status)](https://travis-ci.org/IQSS/dataverse)
+[![Unit Test Coverage](https://img.shields.io/coveralls/github/IQSS/dataverse?label=Unit%20Test%20Coverage)](https://coveralls.io/github/IQSS/dataverse?branch=develop)
 
 [dataverse.org]: https://dataverse.org
 [demo.dataverse.org]: https://demo.dataverse.org

conf/docker-aio/configure_doi.bash

@@ -17,3 +17,8 @@ if [ ! -z "${doi_baseurl}" ]; then
         doi_baseurl_esc=`echo ${doi_baseurl} | sed -e 's/:/\\:/'`
         bin/asadmin create-jvm-options "\"-Ddoi.baseurlstring=${doi_baseurl_esc}\""
 fi
+if [ ! -z "${doi_mdcbaseurl}" ]; then
+        bin/asadmin delete-jvm-options "-Ddoi.mdcbaseurlstring=https\://api.test.datacite.org"
+        doi_mdcbaseurl_esc=`echo ${doi_mdcbaseurl} | sed -e 's/:/\\:/'`
+        bin/asadmin create-jvm-options "\"-Ddoi.mdcbaseurlstring=${doi_mdcbaseurl_esc}\""
+fi

conf/docker-aio/run-test-suite.sh

@@ -8,4 +8,4 @@ fi
 
 # Please note the "dataverse.test.baseurl" is set to run for "all-in-one" Docker environment.
 # TODO: Rather than hard-coding the list of "IT" classes here, add a profile to pom.xml.
-mvn test -Dtest=DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT -Ddataverse.test.baseurl=$dvurl
+mvn test -Dtest=DataversesIT,DatasetsIT,SwordIT,AdminIT,BuiltinUsersIT,UsersIT,UtilIT,ConfirmEmailIT,FileMetadataIT,FilesIT,SearchIT,InReviewWorkflowIT,HarvestingServerIT,MoveIT,MakeDataCountApiIT,FileTypeDetectionIT,EditDDIIT,ExternalToolsIT,AccessIT -Ddataverse.test.baseurl=$dvurl

doc/Architecture/update-user-account-info.png

@@ -72,7 +72,7 @@ Second, file level tools that did not previously define a "contentType" are now
 
 The Dataverse team has been reaching out to tool makers about these breaking changes and getting various tools working in the https://github.com/IQSS/dataverse-ansible repo. Thank you for your patience as the dust settles around the external tool framework.
 
-For more information, check out new <a href=http://guides.dataverse.org/en/4.17/api/external-tools.html/>Building External Tools</a> section of the API Guide.
+For more information, check out new <a href=http://guides.dataverse.org/en/4.17/api/external-tools.html>Building External Tools</a> section of the API Guide.
 
 ## Complete List of Changes
 

doc/release-notes/4.17-release-notes.md

@@ -0,0 +1,118 @@
+# Dataverse 4.18
+
+This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project.
+
+## Release Highlights
+
+### File Page Previews and Previewers
+
+File-level External Tools can now be configured to display in a "Preview Mode" designed for embedding within the file landing page.
+
+While not technically part of this release, previewers have been made available for several common file types. The previewers support for spreadsheet, image, text, document, audio, video, html files and more. These previewers can be found in the <a href="https://github.com/QualitativeDataRepository/dataverse-previewers">Qualitative Data Repository Github Repository</a>.
+
+### Microsoft Login
+
+Users can now create Dataverse accounts and login using self-provisioned Microsoft accounts such as live.com and outlook.com. Users can also use Microsoft accounts managed by their institutions. This new feature not only makes it easier to log in to Dataverse but will also streamline the interaction between any external tools that utilize Azure services that require login.
+
+### Add Data and Host Dataverse
+
+More workflows to add data have been added across the UI, including a new button on the My Data tab of the Account page, as well as a link in the Dataverse navbar, which will display on every page. This will provider users much easier access to start depositing data. By default, the Host Dataverse will be the installation root dataverse for these new Add Data workflows, but there is now a dropdown component allowing creators to select a dataverse you have proper permissions to create a new dataverse or dataset in.
+
+### Primefaces 7
+
+Primefaces, the open source UI framework upon which the Dataverse front end is built, has been updated to the most recent version. This provides security updates and bug fixes and will also allow Dataverse developers to take advantage of new features and enhancements.
+
+### Integration Test Pipeline and Test Health Reporting
+
+As part of the Dataverse Community's ongoing efforts to provide more robust automated testing infrastructure, and in support of the project's desire to have the develop branch constantly in a "release ready" state, API-based integration tests are now run every time a branch is merged to develop. The status of the last test run is available as a badge at the bottom of the README.md file that serves as the homepage of Dataverse Github Repository.
+
+### Make Data Count Metrics Updates
+
+A new configuration option has been added that allows Make Data Count metrics to be collected, but not reflected in the front end. This option was designed to allow installations to collect and verify metrics for a period before turning on the display to users.
+
+### Search API Enhancements
+
+The Dataverse Search API will now display unpublished content when an API token is passed (and appropriate permissions exist).
+
+### Additional Dataset Author Identifiers
+
+The following dataset author identifiers are now supported:
+
+- DAI: https://en.wikipedia.org/wiki/Digital_Author_Identifier
+- ResearcherID: http://researcherid.com
+- ScopusID: https://www.scopus.com
+
+## Major Use Cases
+
+Newly-supported use cases in this release include:
+
+- Users can view previews of several common file types, eliminating the need to download or explore a file just to get a quick look.
+- Users can log in using self-provisioned Microsoft accounts and also can log in using Microsoft accounts managed by an organization.
+- Dataverse administrators can now revoke and regenerate API tokens with an API call.
+- Users will receive notifications when their ingests complete, and will be informed if the ingest was a success or failure.
+- Dataverse developers will receive feedback about the health of the develop branch after their pull request was merged.
+- Dataverse tool developers will be able to query the Dataverse API for unpublished data as well as published data.
+- Dataverse administrators will be able to collect Make Data Count metrics without turning on the display for users.
+- Users with a DAI, ResearcherID, or ScopusID and use these author identifiers in their datasets.
+
+## Notes for Dataverse Installation Administrators
+
+### API Token Management
+
+- You can now delete a user's API token, recreate a user's API token, and find a token's expiration date. See the <a href="http://guides.dataverse.org/en/4.18/api/native-api.html">Native API guide</a> for more information.
+
+### New JVM Options
+
+[:mdcbaseurlstring](http://guides.dataverse.org/en/4.18/installation/config.html#mdcbaseurlstring) allows dataverse administrators to use a test base URL for Make Data Count.
+
+### New Database Settings
+
+[:DisplayMDCMetrics](http://guides.dataverse.org/en/4.18/installation/config.html#DisplayMDCMetrics) can be set to false to disable display of MDC metrics.
+
+## Notes for Tool Developers and Integrators
+
+### Preview Mode
+
+Tool Developers can now add the `hasPreviewMode` parameter to their file level external tools. This setting provides an embedded, simplified view of the tool on the file pages for any installation that installs the tool. See <a href="http://guides.dataverse.org/en/4.18/api/external-tools.html">Building External Tools</a> for more information.
+
+### API Token Management
+
+If your tool writes content back to Dataverse, you can now take advantage of administrative endpoints that delete and re-create API tokens. You can also use an endpoint that provides the expiration date of a specific API token. See the <a href="http://guides.dataverse.org/en/4.18/api/native-api.html">Native API guide</a> for more information.
+
+### View Unpublished Data Using Search API
+
+If you pass a token, the search API output will include unpublished content.
+
+## Complete List of Changes
+
+For the complete list of code changes in this release, see the <a href="https://github.com/IQSS/dataverse/milestone/85?closed=1">4.18 milestone</a> in Github.
+
+For help with upgrading, installing, or general questions please post to the <a href="https://groups.google.com/forum/#!forum/dataverse-community">Dataverse Google Group</a> or email [email protected].
+
+## Installation
+
+If this is a new installation, please see our <a href="http://guides.dataverse.org/en/4.18/installation/">Installation Guide</a>.
+
+## Upgrade
+
+1. Undeploy the previous version.
+
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin list-applications
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin undeploy dataverse
+
+2. Stop glassfish and remove the generated directory, start.
+
+- service glassfish stop
+- remove the generated directory: rm -rf &lt;glassfish install path&gt;glassfish4/glassfish/domains/domain1/generated
+- service glassfish start
+
+3. Deploy this version.
+
+- &lt;glassfish install path&gt;/glassfish4/bin/asadmin deploy &lt;path&gt;dataverse-4.18.war
+
+4. Restart glassfish.
+
+5. Update Citation Metadata Block
+
+- `wget https://github.com/IQSS/dataverse/releases/download/v4.18/citation.tsv`
+- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"`

doc/release-notes/4.18-release-notes.md

@@ -0,0 +1,8 @@
+{
+    "id":"microsoft",
+    "factoryAlias":"oauth2",
+    "title":"Microsoft",
+    "subtitle":"",
+    "factoryData":"type: microsoft | userEndpoint: NONE | clientId: FIXME | clientSecret: FIXME",
+    "enabled":true
+}

doc/sphinx-guides/source/_static/installation/files/root/auth-providers/microsoft.json

@@ -37,6 +37,8 @@ In the curl command below, replace the placeholder "fabulousFileTool.json" place
 
   curl -X POST -H 'Content-type: application/json' http://localhost:8080/api/admin/externalTools --upload-file fabulousFileTool.json 
 
+Note that some tools will provide a preview mode, which provides an embedded, simplified view of the tool on the file pages of your installation. This is controlled by the `hasPreviewMode` parameter. 
+
 Listing All External Tools in Dataverse
 +++++++++++++++++++++++++++++++++++++++
 
@@ -76,10 +78,15 @@ Once you have added an external tool to your installation of Dataverse, you will
 File Level Explore Tools
 ++++++++++++++++++++++++
 
-File level explore tools are specific to the file type (content type or MIME type) of the file. For example, there is a tool for exploring PDF files in the "File Previewers" set of tools.
+File level explore tools are specific to the file type (content type or MIME type) of the file. For example, Data Explorer is tool for exploring tabular data files.
 
 An "Explore" button will appear (on both the dataset page and the file landing page) for files that match the type that the tool has been built for. When there are multiple explore tools for a filetype, the button becomes a dropdown.
 
+File Level Preview Tools
+++++++++++++++++++++++++
+
+File level explore tools can be set up to display in preview mode, which is a simplified view of an explore tool designed specifically for embedding in the file page. 
+
 File Level Configure Tools
 ++++++++++++++++++++++++++
 

doc/sphinx-guides/source/admin/external-tools.rst

@@ -90,7 +90,7 @@ Dataverse supports a protocol called OAI-PMH that facilitates harvesting dataset
 SHARE
 +++++
 
-`SHARE <http://www.share-research.org>`_ is building a free, open, data set about research and scholarly activities across their life cycle. It's possible to add and installation of Dataverse as one of the `sources <https://share.osf.io/sources>`_ they include if you contact the SHARE team.
+`SHARE <http://www.share-research.org>`_ is building a free, open, data set about research and scholarly activities across their life cycle. It's possible to add an installation of Dataverse as one of the `sources <https://share.osf.io/sources>`_ they include if you contact the SHARE team.
 
 Research Data Preservation
 --------------------------

doc/sphinx-guides/source/admin/integrations.rst

@@ -63,8 +63,15 @@ Enable Logging for Make Data Count
 
 To make Dataverse log dataset usage (views and downloads) for Make Data Count, you must set the ``:MDCLogPath`` database setting. See :ref:`MDCLogPath` for details.
 
+If you wish to start logging in advance of setting up other components, or wish to log without display MDC metrics for any other reason, you can set the optional ``:DisplayMDCMetrics`` database setting to false. See :ref:`DisplayMDCMetrics` for details.
+
 After you have your first day of logs, you can process them the next day.
 
+Enable or Disable Display of Make Data Count Metrics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default, when MDC logging is enabled (when ``:MDCLogPath`` is set), Dataverse will display MDC metrics instead of it's internal (legacy) metrics. You can avoid this (e.g. to collect MDC metrics for some period of time before starting to display them) by setting ``:DisplayMDCMetrics`` to false.
+
 Configure Counter Processor
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -128,11 +135,15 @@ Once you are satisfied with your testing, you should contact [email protected]
 - Views ("investigations" in COUNTER)
 - Downloads ("requests" in COUNTER)
 
+.. _MDC-updateCitationsForDataset:
+
 Configuring Dataverse for Make Data Count Citations
 ---------------------------------------------------
 
 Please note: as explained in the note above about limitations, this feature is not available to installations of Dataverse that use Handles.
 
+To configure Dataverse to pull citations from the test vs. production DataCite server see :ref:`doi.mdcbaseurlstring` in the Installation Guide.
+
 Please note that in the curl example, Bash environment variables are used with the idea that you can set a few environment variables and copy and paste the examples as is. For example, "$DOI" could become "doi:10.5072/FK2/BL2IBM" by issuing the following export command from Bash:
 
 ``export DOI="doi:10.5072/FK2/BL2IBM"``