Skip to content

Commit

Permalink
Prepare version 1.11.0 (#98)
Browse files Browse the repository at this point in the history
* Sync with hotfix 1.9.2 (#69)

* Hotfix 1.9.1 (#66)

* Fix RST syntax error

* Remove orphaned print statement

* Update CL

* Bump version 1.9.1 -> 1.9.2

Co-authored-by: Steffengreiner <Steffen.Greiner@gmx.de>

* Update maxquant file parser (#74)

The summary file is now searched within the summary folder

* Remove qube files and Bump version to 1.11.0-SNAPSHOT

* Bump conf.py to current version

* Bump version to 1.11.0-SNAPSHOT

* Include groovydoc plugin in report generation (#59)

* Feature/update nanopore (#91)

* increase some versions, fix testing env

* add tests for alternative nanopore structure

* Add worflow triggers (#93)

* Add triggers

* Reorder git commands

* Use master as base, not main

* parser

* update workflows (#94)

* Set version to 1.10.6

* update pom

* Fix project setup (#96)

* add possibility of some metadata values being empty

* comments

* extract method to validate a single schema

Co-authored-by: Sven F <sven.fillinger@qbic.uni-tuebingen.de>
Co-authored-by: JohnnyQ5 <support@qbic.zendesk.com>
Co-authored-by: Steffengreiner <Steffen.Greiner@gmx.de>
Co-authored-by: Sven F <9976560+sven1103@users.noreply.github.com>

* Feature/update nanopore (#97)

* increase some versions, fix testing env

* add tests for alternative nanopore structure

* Add worflow triggers (#93)

* Add triggers

* Reorder git commands

* Use master as base, not main

* parser

* update workflows (#94)

* Set version to 1.10.6

* update pom

* Fix project setup (#96)

* add possibility of some metadata values being empty

* comments

* extract method to validate a single schema

* use data-model-lib release

Co-authored-by: Sven F <sven.fillinger@qbic.uni-tuebingen.de>
Co-authored-by: JohnnyQ5 <support@qbic.zendesk.com>
Co-authored-by: Steffengreiner <Steffen.Greiner@gmx.de>
Co-authored-by: Sven F <9976560+sven1103@users.noreply.github.com>

* Update CHANGELOG.rst

Co-authored-by: Tobias Koch <KochTobi@users.noreply.github.com>

Co-authored-by: Sven F <sven.fillinger@qbic.uni-tuebingen.de>
Co-authored-by: Steffengreiner <Steffen.Greiner@gmx.de>
Co-authored-by: jnnfr <jennifer.boedker@qbic.uni-tuebingen.de>
Co-authored-by: JohnnyQ5 <support@qbic.zendesk.com>
Co-authored-by: Sven F <9976560+sven1103@users.noreply.github.com>
Co-authored-by: Tobias Koch <KochTobi@users.noreply.github.com>
  • Loading branch information
7 people authored May 6, 2022
1 parent 764885e commit 3d701e4
Show file tree
Hide file tree
Showing 63 changed files with 241 additions and 15 deletions.
5 changes: 5 additions & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ Changelog

This project adheres to `Semantic Versioning <https://semver.org/>`_.

For information about newer versions please refer to `our GitHub releases <https://github.com/qbicsoftware/core-utils-lib/releases>`_

1.10.4 (2022-01-04)
------------------

Expand Down Expand Up @@ -105,6 +107,7 @@ This project adheres to `Semantic Versioning <https://semver.org/>`_.

**Deprecated**


1.9.2 (2021-07-20)
------------------

Expand All @@ -114,6 +117,8 @@ This project adheres to `Semantic Versioning <https://semver.org/>`_.

* Removes orphaned log statement

* Update 'life.qbic.utils.MaxQuantParser', parsing the summary file from a new summary folder (`#74 <https://github.com/qbicsoftware/core-utils-lib/pull/74>`_)

**Dependencies**

**Deprecated**
Expand Down
4 changes: 2 additions & 2 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@
# the built documents.
#
# The short X.Y version.
version = '1.10.2'
version = '1.11.0-SNAPSHOT'
# The full version, including alpha/beta/rc tags.
release = '1.10.2'
release = '1.11.0-SNAPSHOT'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
Expand Down
20 changes: 15 additions & 5 deletions pom.xml
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
<?xml version="1.0" encoding="UTF-8"?>
<project
xmlns="http://maven.apache.org/POM/4.0.0">
xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<groupId>life.qbic</groupId>
<artifactId>core-utils-lib</artifactId>
<description>Collection of non-Vaadin, non-Liferay utilities.</description>
<modelVersion>4.0.0</modelVersion>
<name>Core Utilities Library</name>
<version>1.10.6</version>
<version>1.11.0-SNAPSHOT</version>
<url>https://github.com/qbicsoftware/core-utils-lib</url>
<packaging>jar</packaging>
<properties>
Expand Down Expand Up @@ -125,7 +127,7 @@
<dependency>
<artifactId>data-model-lib</artifactId>
<groupId>life.qbic</groupId>
<version>2.16.1</version>
<version>2.18.0</version>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
Expand Down Expand Up @@ -213,12 +215,12 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
<version>3.7.1</version>
<version>3.9.1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-project-info-reports-plugin</artifactId>
<version>3.0.0</version>
<version>3.1.1</version>
</plugin>
<plugin>
<groupId>life.qbic</groupId>
Expand Down Expand Up @@ -249,4 +251,12 @@
</plugin>
</plugins>
</build>
<reporting>
<plugins>
<plugin>
<groupId>life.qbic</groupId>
<artifactId>groovydoc-maven-plugin</artifactId>
</plugin>
</plugins>
</reporting>
</project>
29 changes: 21 additions & 8 deletions src/main/groovy/life/qbic/utils/NanoporeParser.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package life.qbic.utils
import com.fasterxml.jackson.databind.ObjectMapper
import groovy.json.JsonSlurper
import life.qbic.datamodel.instruments.OxfordNanoporeInstrumentOutput
import life.qbic.datamodel.instruments.OxfordNanoporeInstrumentOutputV2
import org.everit.json.schema.Schema
import org.everit.json.schema.ValidationException
import org.everit.json.schema.loader.SchemaLoader
Expand Down Expand Up @@ -38,7 +39,7 @@ class NanoporeParser {
} catch (ValidationException validationException) {
// we have to fetch all validation exceptions
def causes = validationException.getAllMessages().collect{ it }.join("\n")
throw validationException
throw new ValidationException(causes)
}
}

Expand Down Expand Up @@ -87,12 +88,13 @@ class NanoporeParser {
jsonEnded = true
}
}
def finalMetaData = (Map) jsonSlurper.parseText(buffer.toString())

def finalMetaData = (Map) jsonSlurper.parseText(buffer.toString())
new File(Paths.get(root.toString(), summaryFile["path"].toString()) as String)
.readLines().each { line ->
def split = line.split("=")
finalMetaData[split[0]] = split[1]
def value = split.size() > 1 ? split[1] : ""
finalMetaData[split[0]] = value
}

return finalMetaData
Expand Down Expand Up @@ -155,15 +157,26 @@ class NanoporeParser {
* @throws org.everit.json.schema.ValidationException
*/
private static void validateJson(String json) throws ValidationException {
// Step1: load schema
// Step 1: load schema
JSONObject jsonObject = new JSONObject(json)
InputStream schemaStream = OxfordNanoporeInstrumentOutput.getSchemaAsStream()
JSONObject rawSchema = new JSONObject(new JSONTokener(schemaStream))
Schema jsonSchema = SchemaLoader.load(rawSchema)
// Step2: validate against schema return if valid, throw exception if invalid
try {
// Step 2: validate against schema return if valid, throw exception if invalid
validateUsingSchema(OxfordNanoporeInstrumentOutput.getSchemaAsStream(), jsonObject)
} catch (ValidationException validationException) {
validateUsingSchema(OxfordNanoporeInstrumentOutputV2.getSchemaAsStream(), jsonObject)
}
}

private static void validateUsingSchema(InputStream schemaAsStream, JSONObject jsonObject) throws ValidationException {
Schema jsonSchema = loadSchemaFromStream(schemaAsStream)
jsonSchema.validate(jsonObject)
}

private static Schema loadSchemaFromStream(InputStream stream) {
JSONObject rawSchema = new JSONObject(new JSONTokener(stream))
return SchemaLoader.load(rawSchema)
}

/*
* Converts a file tree into a json object.
*/
Expand Down
30 changes: 30 additions & 0 deletions src/test/groovy/life/qbic/utils/NanoporeParserSpec.groovy
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,36 @@ class NanoporeParserSpec extends Specification {
assert experiment.getMeasurements().get(0).getLibraryPreparationKit() == "SQK-LSK109"
}

def "parsing the alternative valid file structure returns an OxfordNanoporeExperiment Object"() {
given:
def pathToDirectory = Paths.get(exampleDirectoriesRoot, "validates/QABCD001AB_E12A345a01_PAE12345_nanopore_new")
when:
def experiment = NanoporeParser.parseFileStructure(pathToDirectory)
then:
assert experiment instanceof OxfordNanoporeExperiment
// Check that the metadata from the report file has been retrieved
assert experiment.getMeasurements().get(0).getMachineHost() == "PCT0094"
// Check that the metadata from the summary file has been retrieved
assert experiment.getMeasurements().get(0).getLibraryPreparationKit() == "SQK-LSK109-XL"
}

def "parsing the alternative valid file structure with metadata missing returns an OxfordNanoporeExperiment Object"() {
given:
def pathToDirectory = Paths.get(exampleDirectoriesRoot, "validates/QABCD001AB_E12A345a01_PAE12345_nanopore_new_minimal")
when:
def experiment = NanoporeParser.parseFileStructure(pathToDirectory)
then:
assert experiment instanceof OxfordNanoporeExperiment
// Check that the metadata from the report file is parsed
assert experiment.getMeasurements().get(0).getFlowCellType() == "FLO-PRO002"
// Check that the metadata from the summary file has been retrieved, but data can also be empty
assert experiment.getMeasurements().get(0).getLibraryPreparationKit() == "SQK-LSK109-XL"
assert experiment.getMeasurements().get(0).getFlowcellId() == "flow_cell_from_summary"
assert experiment.getMeasurements().get(0).getAsicTemp() == ""
assert experiment.getMeasurements().get(0).getMachineHost() == ""
assert experiment.getMeasurements().get(0).getStartDate() == ""
}

def "qc folder is ignored"() {
given:
def pathToDirectory = Paths.get(exampleDirectoriesRoot, "validates/with_qc_folder/QABCD001AB_E12A345a01_PAE12345")
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
instrument=PCT0094
position=1-A3-D3
flow_cell_id=PAE24142
sample_id=QNANO027AE_E19D023a01_200211
protocol_group_id=20200211_QNANO
protocol=sequencing/sequencing_PRO002_DNA:FLO-PRO002:SQK-LSK109-XL
protocol_run_id=5a7cfc2a-81b0-412d-baa0-51b939cd8e76
acquisition_run_id=c6028297dff19d01e7c5fba6487de807d1e99c05
started=2020-02-11T15:52:10.465982+01:00
acquisition_stopped=2020-02-14T08:39:54.688916+01:00
processing_stopped=2020-02-14T08:39:58.804639+01:00
basecalling_enabled=1
sequencing_summary_file=sequencing_summary_PAE24142_c6028297.txt
fast5_files_in_final_dest=2189
fast5_files_in_fallback=0
fastq_files_in_final_dest=2189
fastq_files_in_fallback=0
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
Tracking ID
===========

{
"asic_id": "0004A30B0022C63E",
"asic_id_eeprom": "0004A30B0022C63E",
"asic_temp": "32.631687",
"asic_version": "Unknown",
"auto_update": "0",
"auto_update_source": "https://mirror.oxfordnanoportal.com/software/MinKNOW/",
"bream_is_standard": "0",
"configuration_version": "1.0.7",
"device_id": "1-E9-H9",
"device_type": "promethion",
"distribution_status": "stable",
"distribution_version": "19.12.5",
"exp_script_name": "N/A",
"exp_script_purpose": "sequencing_run",
"exp_start_time": "2020-01-28T15:17:38Z",
"flow_cell_id": "PAE26989",
"flow_cell_product_code": "FLO-PRO002",
"guppy_version": "3.2.8+bd67289",
"heatsink_temp": "36.179111",
"hostname": "PCT0094",
"hublett_board_id": "0132136faade2e15",
"hublett_firmware_version": "2.0.12",
"installation_type": "nc",
"ip_address": "",
"local_firmware_file": "1",
"mac_address": "",
"operating_system": "ubuntu 16.04",
"protocol_group_id": "20200128_QNANO",
"protocol_run_id": "",
"protocols_version": "4.3.16",
"run_id": "db9e9383d44d80bbe1e2600c7a7419056610d46d",
"sample_id": "QNANO036AD_E19D023b04",
"satellite_board_id": "0000000000000000",
"satellite_firmware_version": "2.0.12",
"usb_config": "firm_1.2.3_ware#rbt_4.5.6_rbt#ctrl#USB3",
"version": "3.6.1"
}

Duty Time
=========

ID: db9e9383d44d80bbe1e2600c7a7419056610d46d

Channel State,Experiment Time (minutes),State Time (samples),
strand,0,144832342
strand,1,158421270
strand,2,378095352
strand,3,472685319
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
This is some text
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
instrument=
position=1-A3-D3
flow_cell_id=flow_cell_from_summary
sample_id=QNANO027AE_E19D023a01_200211
protocol_group_id=
protocol=sequencing/sequencing_PRO002_DNA:FLO-PRO002:SQK-LSK109-XL
protocol_run_id=
acquisition_run_id=
started=
acquisition_stopped=2020-02-14T08:39:54.688916+01:00
processing_stopped=2020-02-14T08:39:58.804639+01:00
basecalling_enabled=
sequencing_summary_file=sequencing_summary_PAE24142_c6028297.txt
fast5_files_in_final_dest=2189
fast5_files_in_fallback=0
fastq_files_in_final_dest=2189
fastq_files_in_fallback=0
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
Tracking ID
===========

{
"asic_id": "",
"asic_id_eeprom": "",
"asic_temp": "",
"asic_version": "",
"auto_update": "",
"auto_update_source": "",
"bream_is_standard": "",
"configuration_version": "",
"device_id": "",
"device_type": "promethion",
"distribution_status": "",
"distribution_version": "",
"exp_script_name": "",
"exp_script_purpose": "",
"exp_start_time": "2020-01-28T15:17:38Z",
"flow_cell_id": "flow_cell_from_report",
"flow_cell_product_code": "FLO-PRO002",
"guppy_version": "3.2.8+bd67289",
"heatsink_temp": "",
"hostname": "",
"hublett_board_id": "",
"hublett_firmware_version": "",
"installation_type": "",
"ip_address": "",
"local_firmware_file": "",
"mac_address": "",
"operating_system": "",
"protocol_group_id": "",
"protocol_run_id": "",
"protocols_version": "",
"run_id": "",
"sample_id": "QNANO036AD_E19D023b04",
"satellite_board_id": "",
"satellite_firmware_version": "",
"usb_config": "",
"version": ""
}

Duty Time
=========

ID: db9e9383d44d80bbe1e2600c7a7419056610d46d

Channel State,Experiment Time (minutes),State Time (samples),
strand,0,144832342
strand,1,158421270
strand,2,378095352
strand,3,472685319

0 comments on commit 3d701e4

Please sign in to comment.