contents of a document

[Eran-T]

Hi,
I have a question regarding the contents of a document.

My first questions are:

1. given a docket which has both non-empty docket-entries and opinion-clusters, what would be the difference between the documents found between the two (other than the available fields for each). Is the difference that docket entries may contain documents other than opinions as opposed to opinion clusters?
2. If an opinion's 'plain_text' field is non empty for a certain docket-entry, is that opinion integrated into the opinions endpoint? (meaning, am I guaranteed to find that opinion through the opinions endpoint or is it possible that it will only be available through the docket-entries endpoint) if so, under which field would the 'plain_text' field from the docket-entries endpoint be available in its opinion endpoint counterpart?
3. assuming the following fields all refer to the contents of an opinion "plain_text", "html", "html_lawbox", "html_columbia", "html_anon_2020", "xml_harvard", "html_with_citations". Are they mutually exclusive depending on the source of the opinion? If they are not mutually exclusive, do they represent the same information, or can some fields complement each other? If they represent the same information, are there particular fields that I should prioritize over others given that I seek the most "complete" information? (For example, let's say a document has both a plain_text and an html fields, they both represent the same document however, the process of converting from HTML to plain text tends to cause certain information such as the signatures section to be discarded due to presentation or technical reasons. than I would prefer the html field)

Thank you for your time and help!

[mlissner]

Thank you for your time and help!

We're always happy to answer these questions.

1. given a docket which has both non-empty docket-entries and opinion-clusters, what would be the difference between the documents found between the two (other than the available fields for each). Is the difference that docket entries may contain documents other than opinions as opposed to opinion clusters?

Docket entries could be anything, yes: Filings, opinions, motions, exhibits, whatever. Items in the Opinions table are only...opinions! And yes, it's true that you can find opinions in the docket entries table, but they're different because they don't aim to have the same level of quality. They lack HTML, they don't have granular metadata, etc.

My advice, if you're working with opinions, is steer clear of the docket-entries tables, if you can.

2. If an opinion's 'plain_text' field is non empty for a certain docket-entry, is that opinion integrated into the opinions endpoint? (meaning, am I guaranteed to find that opinion through the opinions endpoint or is it possible that it will only be available through the docket-entries endpoint) if so, under which field would the 'plain_text' field from the docket-entries endpoint be available in its opinion endpoint counterpart?

If you find something in the docket-entries table, I'd ignore it, if you can, and look for a copy of it in the opinions tables.

3. assuming the following fields all refer to the contents of an opinion "plain_text", "html", "html_lawbox", "html_columbia", "html_anon_2020", "xml_harvard", "html_with_citations". Are they mutually exclusive depending on the source of the opinion? If they are not mutually exclusive, do they represent the same information, or can some fields complement each other? If they represent the same information, are there particular fields that I should prioritize over others given that I seek the most "complete" information? (For example, let's say a document has both a plain_text and an html fields, they both represent the same document however, the process of converting from HTML to plain text tends to cause certain information such as the signatures section to be discarded due to presentation or technical reasons. than I would prefer the html field)

We get the content from a variety of sources and when we do we save the various representations of the text that we get into separate fields. In general, the best field to use is html_with_citations, since it has the hyperlinked citations in it and it's a field that we generate based on the best text field we can find.

• If that field is not populated, the best approach is to choose the first fields that is populated, according to the following order (from best to worst):
  • html_with_citations is a special field that is populated by parsing one of the above fields for citations, generating an HTML file with hyperlinked citations. All items should eventually have this field, though it can be empty initially or if the cross-linker crashes. In general, this is the field that is used to generate pages on CourtListener and the one we recommend.
  • html_columbia will be populated if we got the content from the Columbia collaboration.
  • html_lawbox will be populated if we got the content from the Lawbox donation.
  • xml_harvard will be populated if the source was Harvard's Caselaw Access Project. This field has a lot of data but is inferior to others due to being created by OCR instead of by humans.
  • html_anon_2020 will be populated if we got the content from our anonymous source in 2020.
  • html will be populated if we got the opinion from a court's website as a Word Perfect or HTML document, or if we got the opinion from Resource.org, which provides HTML documents.
  • plain_text will be populated if we got the opinion from a court's website as a PDF or Microsoft Word document.

[Eran-T]

Hi, thanks for the detailed reply.
One thing I would like to clarify is: if an opinion is found in the docket-entries will it necessarily have an opinions opinion corresponding to it (corresponding to the same opinion)?

[mlissner]

if an opinion is found in the docket-entries will it necessarily have an opinions opinion corresponding to it

I wish, but there are at least two independent routes to federal content being an "opinion":

1. A clerk uploads an opinion to PACER and remembers to check the box saying it's an opinion. That winds up in the docket-entries API as an opinion.

2. A publisher/judge decides/learns that a document is important and puts it in a book. The way this decision is made is unclear. I've heard judges work directly with the publishers, I've heard publishers make editorial decisions. Etc — but the point is that this content is what most people think of as a "published opinion" and it's what winds up in our opinions database.

So, that leaves four possible outcomes:

• 1: Rule 1 applies and the item is marked as an opinion by the clerk:
  • 1.1: And the publisher agrees! It winds up published and should be in our opinions dataset.
  • 1.2: And the publisher disagrees. It does not wind up published and should not be in our opinions dataset.
• 2: Rule 2 applies, and:
  • 2.1: The judge decides that it should be published, the publisher agrees, it winds up in our opinions dataset.
  • 2.2: The judge says it should be published, but the publisher disagrees, and it does not wind up in our opinions dataset. (I don't know if this actually happens in practice, I sort of doubt it, but perhaps sometimes. I'm just covering all the bases.)

So to answer your question more directly, if something is marked as an opinion in the docket-entries API, it may or may not be considered an important decision and it may or may not wind up with a copy in our opinions database. I think my general advice would be that you can ignore opinions in the docket-entries API, except for some corner cases we're working on pinning down.