Merge pull request #1913 from jairideout/issue-1904

DOC: fix broken links and script RST docs; use help.qiime.org

CONTRIBUTING.md

@@ -1,12 +1,11 @@
 Contributing to QIIME
 =====================
 
-[QIIME](http://www.qiime.org) is an open source software package, and we welcome community contributions. You can find the source code and test code for QIIME under public revision control in the QIIME git repository on [GitHub](https://github.com/qiime/qiime). While we have a core development group, we very much welcome contributions from other users. 
+[QIIME](http://www.qiime.org) is an open source software package, and we welcome community contributions. You can find the source code and test code for QIIME under public revision control in the QIIME git repository on [GitHub](https://github.com/qiime/qiime). While we have a core development group, we very much welcome contributions from other users.
 
 This document covers what you should do to get started with contributing to QIIME. You should read this whole document before considering submitting code to QIIME. This will save time for both you and the QIIME developers.
 
-If you're looking for help with QIIME, you should post an issue to the [QIIME Forum](http://forum.qiime.org).
-
+To get help with using QIIME, please visit http://help.qiime.org.
 
 Type of Submissions
 -------------------
@@ -17,7 +16,7 @@ When considering submitting a new feature to QIIME, you should begin by posting
 
 * For new features, you'll want to describe why the functionality that you are proposing to add is relevant. For it to be relevant, it should be demonstrably useful to QIIME users. This typically means that a new analytic method is implemented (you should describe why it's useful, ideally including a link to a paper that uses this method), or an existing method is enhanced (your implementation matches the performance of the pre-existing method while reducing runtime, memory consumption, etc, or it improves performance over the pre-existing method). We will request benchmark results comparing your method to the pre-existing methods (which would also be required for publication of your method) so pointing to a paper or other document containing benchmark results, or including benchmark results in your issue, will speed up the process.
 
-* For bug fixes, you should provide a detailed description of the bug so other developers can reproduce it. We take bugs in QIIME very seriously. Bugs can be related to errors in code, documentation, or tests. Errors in documentation or tests are usually updated in the next major release of QIIME. Errors in code that could result in incorrect results or inability to access certain functionality may result in a new minor release of QIIME. 
+* For bug fixes, you should provide a detailed description of the bug so other developers can reproduce it. We take bugs in QIIME very seriously. Bugs can be related to errors in code, documentation, or tests. Errors in documentation or tests are usually updated in the next major release of QIIME. Errors in code that could result in incorrect results or inability to access certain functionality may result in a new minor release of QIIME.
 
  You should include the following information in your bug report:
 
@@ -45,15 +44,15 @@ When you submit code to QIIME, it will be reviewed by one or more QIIME develope
 * Your code is sufficiently well-documented (see Coding Guidelines below).
 * Your code provides relevant changes or additions to QIIME (Type of Submissions above).
 
-This process is designed to ensure the quality of QIIME, and can be a very useful experience for new developers. 
+This process is designed to ensure the quality of QIIME, and can be a very useful experience for new developers.
 
 Particularly for big changes, if you'd like feedback on your code in the form of a code review as you work, you should request help in the issue that you created and one of the QIIME developers will work with you to perform regular code reviews. This can greatly reduce development time (and frustration) so we highly recommend that new developers take advantage of this rather than submitting a pull request with a massive amount of code in one chunk. That can lead to frustration when the developer thinks they are done, but the reviewer requests large amounts of changes, and it is also very hard to review.
 
 
 Submitting code to QIIME
 ------------------------
 
-QIIME is hosted on [GitHub](http://www.github.com), and we use GitHub's [Pull Request](https://help.github.com/articles/using-pull-requests) mechanism for accepting submissions. You should go through the following steps to submit code to QIIME. 
+QIIME is hosted on [GitHub](http://www.github.com), and we use GitHub's [Pull Request](https://help.github.com/articles/using-pull-requests) mechanism for accepting submissions. You should go through the following steps to submit code to QIIME.
 
 1. Begin by [creating an issue](https://github.com/qiime/qiime/issues) describing your proposed change. This should include a description of your proposed change (is it a new feature, a bug fix, etc.), and note in the issue description that you want to work on it. If you'll be modifying existing QIIME file(s), you'll want to get input from the developer responsible for the relevant file(s) via a discussion on the issue tracker to let them know you what you'd like to do. The developer responsible for the code is named in the ``__maintainer__`` variable at the top of the file. Once you hear back that it is OK to make changes (i.e., they don't have local edits, they agree with the change you'd like to make, and they're comfortable with you editing their code), we will assign the issue to you on GitHub.
 

ChangeLog.md

@@ -7,12 +7,14 @@ New scripts
 * ``observation_metadata_correlation.py``: Allows the calculation of correlations between feature abundances and continuous-valued metadata. This script replaces the continuous-valued correlation functionality that was in ``otu_category_significance.py`` in QIIME 1.7.0 and earlier.
 * ``compare_trajectories.py``: Allows analysis of volatility using different algorithms.
 * ``compute_taxonomy_ratios.py``: Implements the microbial dysbiosis index (MD-index) from [Gevers et al 2014](http://www.ncbi.nlm.nih.gov/pubmed/24629344).
-* ``collpase_samples.py``: Allows collapsing groups of samples in BIOM tables and mapping files based on their metadata (see [#1678](https://github.com/biocore/qiime/issues/1678)). This can be used, for example, to collapse samples belonging to a replicate group. This also has replaced ``summarize_otu_by_cat.py`` (see discussion on [#1798](https://github.com/biocore/qiime/issues/1798)).
+* ``collapse_samples.py``: Allows collapsing groups of samples in BIOM tables and mapping files based on their metadata (see [#1678](https://github.com/biocore/qiime/issues/1678)). This can be used, for example, to collapse samples belonging to a replicate group. This also has replaced ``summarize_otu_by_cat.py`` (see discussion on [#1798](https://github.com/biocore/qiime/issues/1798)).
 * ``multiple_split_libraries_fastq.py``, ``multiple_join_paired_ends.py``, and ``multiple_extract_barcodes.py``: Facilitate initial QIIME processing of already-demultiplexed fastq files, as these are commonly being provided by sequencing centers.
 * ``differential_abundance.py``: Supplements ``group_significance.py`` to support metagenomeSeq's fitZIG algorithm and DESeq2's negative binomial algorithm.  The input for this is an unnormalized, raw BIOM table.
 * ``normalize_table.py``: Adds support for BIOM table normalization algorithms in addition to rarefaction. Supported methods are metagenomeSeq's CSS and DESeq's variance stabilizing transformation.
 * ``start_parallel_jobs_slurm.py``: Allows for parallel job submission using [slurm](https://computing.llnl.gov/linux/slurm/).
 * ``split_libraries_lea_seq.py``: Allows for demultiplexing of sequences using the LEA-Seq protocol, described in [Faith et al. (2013)](http://www.sciencemag.org/content/341/6141/1237439). This script should be considered to be in **beta testing status**.
+* ``extract_reads_from_interleaved_file.py``: Splits an interleaved FASTQ file (like the ones produced by JGI) into forward and reverse reads. See [this section](http://qiime.org/tutorials/processing_illumina_data.html#processing-joint-genome-institute-fastq-files) of the Illumina data preparation tutorial for more details.
+* ``parallel_pick_otus_sortmerna.py``: Perform parallel OTU picking with SortMeRNA ([Kopylova et al. (2012)](http://www.ncbi.nlm.nih.gov/pubmed/23071270).
 
 Features
 --------

README.md

@@ -3,8 +3,8 @@ QIIME: Quantitative Insights Into Microbial Ecology
 
 [![Build Status](http://ci.qiime.org/job/QIIME/badge/icon)](http://ci.qiime.org/job/QIIME/)
 
-The official QIIME source code repository. For details on QIIME, see www.qiime.org. 
+The official QIIME source code repository. For details on QIIME, see www.qiime.org.
 
 See the [Biocore GitHub organization](https://github.com/biocore) for related software projects and data.
 
-For questions on QIIME, head to the [QIIME Forum](https://groups.google.com/forum/#!forum/qiime-forum)
+To get help with using QIIME, please visit http://help.qiime.org.

doc/documentation/file_formats.rst

@@ -29,7 +29,7 @@ The mapping file relates barcodes in the FASTA file to each sample and their rel
 
 Each column header MUST contain alphanumeric (a-z, A-Z and 1-9) and/or underscore ("_") characters only, where the header MUST start with letter. All other characters (e.g. $, \*, \^, etc) are not supported at this time and use of those characters may cause problems downstream in the QIIME pipeline.
 
-Currently, the user has the ability to define their own column headers, however; QIIME will be adopting the MIMARKS standard, therefore all column headings MUST correspond the proper MIMARKS nomenclature (http://gensc.org/gc_wiki/index.php/MIMARKS). The following details the current mapping file guidelines:
+Currently, the user has the ability to define their own column headers, however; QIIME will be adopting the MIMARKS standard, therefore all column headings MUST correspond the proper MIMARKS nomenclature (http://wiki.gensc.org/index.php?title=MIMARKS). The following details the current mapping file guidelines:
 
 1. The first column header must be "#SampleID", and the data in this column must contain unique (short and meaningful) sample identifiers containing only alphanumeric and period (".") characters. Leading and trailing spaces will raise a warning when using `validate_mapping_file.py <../scripts/validate_mapping_file.html>`_.
 2. The second column header must be "BarcodeSequence", where each value in that column corresponds to the barcode used for each sample.  Only IUPAC DNA characters are acceptable. Leading and trailing spaces will raise a warning when using `validate_mapping_file.py <../scripts/validate_mapping_file.html>`_.

doc/index.rst

@@ -13,7 +13,7 @@ QIIME (canonically pronounced 'Chime') is a pipeline for performing microbial co
 
 Rather than reimplementing commonly used algorithms, QIIME wraps popular implementations of those algorithms. This allows us to make use of the many excellent tools available in this area, and allows faster integration of new tools. If you use tools that you think would be useful additions to QIIME, consider submitting a feature request by `creating a new issue on our GitHub issue tracker <https://github.com/biocore/qiime/issues>`_ and labelling it as an *enhancement*. You will need to create a free GitHub account in order to create an issue.
 
-A standard QIIME analysis begins with sequence data from one or more sequencing platforms, including Sanger, Roche/454, and Illumina GAIIx. QIIME can perform library de-multiplexing and quality filtering; denoising with AmpliconNoise or the QIIME Denoiser; OTU and representative set picking with uclust, cdhit, mothur, BLAST, or other tools; taxonomy assignment with uclust, BLAST, the RDP classifier, mothur, or rtax; sequence alignment with PyNAST, muscle, infernal, or other tools; phylogeny reconstruction with FastTree, raxml, clearcut, or other tools; alpha diversity and rarefaction, including visualization of results, using over 20 metrics including Phylogenetic Diversity, chao1, and observed species; beta diversity and rarefaction, including visualization of results, using over 25 metrics including weighted and unweighted UniFrac, Euclidean distance, and Bray-Curtis; summarization and visualization of taxonomic composition of samples using area, bar and pie charts along with distance histograms; and many other features. While QIIME is primarily used for analysis of amplicon data, many of the downstream analysis pipeline (such as alpha rarefaction and jackknifed beta diversity) can be performed on any type of `sample x observation tables if they are formatted correctly <./documentation/file_formats.html#otu-table-format>`_. 
+A standard QIIME analysis begins with sequence data from one or more sequencing platforms, including Sanger, Roche/454, and Illumina GAIIx. QIIME can perform library de-multiplexing and quality filtering; denoising with AmpliconNoise or the QIIME Denoiser; OTU and representative set picking with uclust, cdhit, mothur, BLAST, or other tools; taxonomy assignment with uclust, BLAST, the RDP classifier, mothur, or rtax; sequence alignment with PyNAST, muscle, infernal, or other tools; phylogeny reconstruction with FastTree, raxml, clearcut, or other tools; alpha diversity and rarefaction, including visualization of results, using over 20 metrics including Phylogenetic Diversity, chao1, and observed species; beta diversity and rarefaction, including visualization of results, using over 25 metrics including weighted and unweighted UniFrac, Euclidean distance, and Bray-Curtis; summarization and visualization of taxonomic composition of samples using area, bar and pie charts along with distance histograms; and many other features. While QIIME is primarily used for analysis of amplicon data, many of the downstream analysis pipeline (such as alpha rarefaction and jackknifed beta diversity) can be performed on any type of `sample x observation tables if they are formatted correctly <./documentation/file_formats.html#otu-table-format>`_.
 
 QIIME includes parallelization capabilities for many of the computationally intensive steps. By default, these are configured to utilize a mutli-core environment, and are easily configured to run in a cluster environment. QIIME is built in Python using the open-source PyCogent_ toolkit. It makes extensive use of unit tests, and is highly modular to facilitate custom analyses.
 
@@ -24,7 +24,7 @@ We recommend that all QIIME users keep an eye on the QIIME blog for important an
 Downloading and installing QIIME
 ================================
 
-There are several ways to get a working install of QIIME. 
+There are several ways to get a working install of QIIME.
 
 The easiest is to use our virtual machines:
 
@@ -35,11 +35,11 @@ The easiest is to use our virtual machines:
 Alternatively, there are a few choices for automated installation of QIIME.
 
  * qiime-deploy: The `qiime-deploy <https://github.com/qiime/qiime-deploy>`_ tool can be used to easily install various versions of QIIME natively on Linux systems.
- * MacQIIME: `MacQIIME <http://www.wernerlab.org/software/macqiime>`_ includes a nearly-full installation of QIIME for Mac OS X users. 
+ * MacQIIME: `MacQIIME <http://www.wernerlab.org/software/macqiime>`_ includes a nearly-full installation of QIIME for Mac OS X users.
 
-If you are instead interested in a native install of QIIME, you can choose between the QIIME base install, which provides access to the most commonly used features of QIIME (with default parameters), or the QIIME full install. For both, you should begin with the `QIIME installation guide <./install/install.html>`_. For most users, the QIIME base install will be sufficient, at least when getting started. 
+If you are instead interested in a native install of QIIME, you can choose between the QIIME base install, which provides access to the most commonly used features of QIIME (with default parameters), or the QIIME full install. For both, you should begin with the `QIIME installation guide <./install/install.html>`_. For most users, the QIIME base install will be sufficient, at least when getting started.
 
-If installing QIIME natively, you'll need to choose between the release and development version of QIIME. 
+If installing QIIME natively, you'll need to choose between the release and development version of QIIME.
 
  * Stable Release: Currently the most stable version of QIIME is our |release| release, which you can `download here <https://pypi.python.org/pypi/qiime>`_. If you don't know whether you want to install the release or development version of QIIME, you most likely want the release version.
 
@@ -54,26 +54,24 @@ After installing QIIME, you should move on to the `QIIME Illumina Overview Tutor
 
 Contact Us
 ==========
-The only way to get help with QIIME is to search or ask questions in the `QIIME Forum <http://groups.google.com/group/qiime-forum>`_. This is a public space for users to ask questions, search previous questions and answers, and share any tips they've found. The QIIME Developers moderate this forum, and we aim to respond to questions within one working day.
-
-Before requesting help with QIIME `review this discussion <http://groups.google.com/group/qiime-forum/browse_thread/thread/67e2ab4af75655ae>`_ to get information on what data to send to help us answer your question as quickly as possible.
+To get help with using QIIME, please visit http://help.qiime.org.
 
 Users can also submit bug reports and feature requests via our `GitHub issue tracker <https://github.com/biocore/qiime/issues>`_.
 
 QIIME Development
 =================
 
-QIIME is an open-source project, primarily developed in the `Knight Lab <https://knightlab.colorado.edu>`_ at the University of Colorado at Boulder, the `Caporaso Lab <http://caporasolab.us>`_ at Northern Arizona University and the `Clemente Lab <http://clemente-lab.github.io>`_ at Icahn School of Medicine at Mount Sinai. If you are interested in getting involved, check out the `developer notes <./developer/index.html>`_.
+QIIME is an open-source project, primarily developed in the `Knight Lab <http://www.knightlab.ucsd.edu>`_ at the University of Colorado at Boulder, the `Caporaso Lab <http://caporasolab.us>`_ at Northern Arizona University and the `Clemente Lab <http://clemente-lab.github.io>`_ at Icahn School of Medicine at Mount Sinai. If you are interested in getting involved, check out the `developer notes <./developer/index.html>`_.
 
 Citing QIIME
 ============
 If you use QIIME for any published research, please include the following citation:
 
 	**QIIME allows analysis of high-throughput community sequencing data**
-	
+
 	J Gregory Caporaso, Justin Kuczynski, Jesse Stombaugh, Kyle Bittinger, Frederic D Bushman, Elizabeth K Costello, Noah Fierer, Antonio Gonzalez Pena, Julia K Goodrich, Jeffrey I Gordon, Gavin A Huttley, Scott T Kelley, Dan Knights, Jeremy E Koenig, Ruth E Ley, Catherine A Lozupone, Daniel McDonald, Brian D Muegge, Meg Pirrung, Jens Reeder, Joel R Sevinsky, Peter J Turnbaugh, William A Walters, Jeremy Widmann, Tanya Yatsunenko, Jesse Zaneveld and Rob Knight; Nature Methods, 2010; doi:10.1038/nmeth.f.303
 
 
-You can find the `QIIME paper here <http://www.nature.com/nmeth/journal/vaop/ncurrent/full/nmeth.f.303.html>`_, and the data presented in this paper can be found `here <http://bmf.colorado.edu/QIIME/QIIME_NM_2010.tgz>`_.
+You can find the `QIIME paper here <http://www.nature.com/nmeth/journal/v7/n5/full/nmeth.f.303.html>`_, and the data presented in this paper can be found `here <ftp://ftp.microbio.me/qiime/QIIME_NM_2010.tgz>`_.
 
 .. _PyCogent: http://www.pycogent.org