Fetching documents in bulk from RECAP?

[amluto]

Hi all-

I want to fetch all documents associated with a docket from PACER and contribute it to RECAP (and I'm happy to pay for it!). After getting some hints from @mlissner, I've gotten this far:

#!/usr/bin/python3

import sys
import requests
import getpass

courtlistener_auth = None

def ask_for_creds():
    username = getpass.getpass("Courtlistener username: ")
    password = getpass.getpass("Courtlistener password: ")

    global courtlistener_auth
    courtlistener_auth = (username, password)

def query_missing_docs(docket_id, start, end):
    url = 'https://www.courtlistener.com/api/rest/v3/docket-entries/'

    # Setting 'recap_documents__is_available': 'false' ends up giving a cartesian product,
    # not the list of docket-entries that have at least one unavailable recap document.

    params = {'docket__id' : ('%d' % docket_id),
              'entry_number__range' : ('%d,%d' % (start, end))}
    response = requests.get(url, params=params, auth=courtlistener_auth)

    if response.status_code != requests.codes.ok:
        response.raise_for_status()

    json = response.json()
    results = json['results']

    # I'm lazy: I'm not handling pagination.  If you ask for too many documents,
    # some will be ignored.
    return [doc for result in results for doc in result['recap_documents'] if doc['is_available'] == False]

def main():
    if len(sys.argv) != 4:
        print('Usage: %s docket_id start end' % sys.argv[0], file=sys.stderr)
        return 1
    docket_id, start, end = sys.argv[1:4]
    
    ask_for_creds()
    docs = query_missing_docs(int(docket_id), int(start), int(end))

    for doc in docs:
        print('Missing doc %s: %s' % (doc['document_number'], doc['description']))

    while True:
        yn = input('Buy them? ')
        if yn.lower() in ('y', 'yes', 'true'):
            break
        elif yn.lower() in ('n', 'no', 'false'):
            return 0

    pacer_username = getpass.getpass('PACER username: ')
    pacer_password = getpass.getpass('PACER password: ')

    for doc in docs:
        # TODO: we want the document (type 2), but we also want to make sure that
        # RECAP has the attachment, if any (type 3), but I don't really understand
        # the attachment documentation.  Hints?
        #
        # It would be nifty if there was a request_type to ask for the document
        # if it's just one document and the attachment list if it's not.
        data = {'request_type':'2',
                'pacer_username' : pacer_username,
                'pacer_password' : pacer_password,
                'recap_document' : doc['id']}
        
        url = "https://www.courtlistener.com/api/rest/v3/recap-fetch/"
        response = requests.post(url, data=data, auth=courtlistener_auth)
        print('Request for doc %s: %d' % (doc['document_number'], response.status_code))
        print(response.text)

if __name__ == '__main__':
    sys.exit(main())

And I have a couple minor issues so far:

• recap_documents__is_available==false works very poorly. If there's a document with two attachments, and all three of the docs in question are unavailable, I get three duplicate docket-entries. I assume I'm querying Django wrong, but I would have expected Django to give an error instead of messing it up MySQL-style. I worked around it.
• Some of my recap-fetch requests are ending up taking quite a bit longer than expected (which is not the end of the world).

And a major issue: I'm doing request-type 2 requests, which works fine for documents where the attachment list is already in RECAP and for documents that don't have attachments, but it does not work for documents that have attachments unknown to RECAP. And I don't know how to distinguish these cases. Any hints?

A request-type to ask for a document if it's a document and the attachment list if it's not (which is what I get if I just click 'Buy on PACER' would help. So would some way to distinguish what's going on from the docket-entry result.

Also, could Courtlistener maybe add a web UI for this? Maybe with a hint that anyone who's about to spend a couple hundred dollars downloading a docket might also donate some money to courtlistener?

[mlissner]

Thanks for posting. A couple thoughts:

1. Your first step, if you want to be thorough, is to ask for the attachment page for any non-minute entry (i.e., ones that have entry numbers. Just do a quick loop through all the items, and get the attachment pages. There's no harm in doing so (or cost), and then you'll have the full universe of content for the case.

3.Once you've done that, you should filter the recap-document API, not the docket entry API, and just loop through those items. I think if you do that, you'll fix your is_available=false problem.

The Web UI idea is a good one. I'd love a one-click button that just downloaded a whole case. Buuuuut, we haven't gotten there yet!

[amluto]

I'm a bit confused. If I understood correctly, there are two kinds of docket-entries, those with entry_number null and those with it non-null. The ones with entry_number == null are "minute entries", right? I found one, and it had a single recap-document associated with it. I sent a request_type==3 fetch for it, and I got:

Unable to get attachment page: Unknown pacer_doc_id for RECAP Document object ...

Help? I'm sure I'm misunderstanding something here.

[mlissner]

there are two kinds of docket-entries, those with entry_number null and those with it non-null.

Correct.

The ones with entry_number == null are "minute entries", right?

Right, so as far as I know, those don't have attachment pages.

What you want to do is loop through all the non-minute docket entries and get their attachment pages. Then go through all the recap-documents and try to download their documents. I think that should work pretty well.

[amluto]

I read through some of the underlying code, and as far as I can see, it's not actually possible to example a DocketEntry with a single RecapDocument and tell whether the attachment page has ever been processed. See here:

courtlistener/cl/recap/mergers.py

Line 1331 in f242381

def merge_attachment_page_data(

The processing of an attachment page with no attachments doesn't appear to do anything to the DocketEntry or to anything else in the database.

Ordinarily (in the absence of errors and in the absence of anyone using recap-fetch), this is more or less okay: there won't be any DocketEntries that have an available RecapDocument but don't have the attachment page merged in. So either users see a single "Buy on PACER" link or they see the full list of attachments with the option to buy any missing ones.

But recap-fetch breaks this:

1. Users can't tell whether they should sent request_type=3, because there is no query that will tell them whether it has been done. Maybe they can remember whether they've already done it.
2. Users (like my script!) that send request_type=2 blindly will create is_available=true RecapDocuments without getting the attachment page, and now even web users who look at the case will be given no indication that an attachment might be missing.

I think the information could be fished out of PacerHtmlFiles by looking for UPLOAD_TYPE.ATTACHMENT_PAGE, but I don't think that's practical for users like me. It's certainly not a documented reasonable thing to do. And the web UI wouldn't want to do that.

Should I open an issue about this? It could be fixed (with some effort for back-populating the data) by adding a field to DocketEntry indicating whether (and from when?) an attachment page is present.

[mlissner]

I haven't seen this as a huge issue. I think what folks usually do is just get the attachment page whenever there is a new docket entry. We've considered being more aggressive/proactive about getting these pages in the past, but it'd require something like 200M requests to backfill them.

[amluto]

The plot thickens a bit. As far as I can tell from brief inspection, request_type==3 requests for documents that actually have attachments work. requests for documents that don't have attachments fail with Not a valid attachment page upload.

I think it's unfortunate that by malice or merely by accident (such as running the original version of my script) load data into RECAP that obscures the potential existence of attachments.

But at least I did get request_type == working, and I'm running a little throttled job to backfill this data for the case that I've been querying.

[mlissner]

Sorry, I don't follow. How does running your script "obscures the potential existence of attachments"?

[amluto]

If I open a docket in a browser, e.g. https://www.courtlistener.com/docket/60120427/us-dominion-inc-v-herring-networks-inc/ (hey, these cases were in the news!), I see three general categories of numbered docket entries:

1. Entries with no documents available. There's a single white "Buy on PACER" button. This means that RECAP doesn't know whether there are attachments. This is fine.
2. Entries that show attachments. (e.g. https://www.courtlistener.com/docket/60120427/us-dominion-inc-v-herring-networks-inc/?filed_after=&filed_before=&entry_gte=1&entry_lte=1&order_by=asc right now.) There's a main entry and there are 6 attachments. Some are blue "Download PDF" buttons and some are white "Buy on PACER" buttons, of which some show a price. This seems fine -- I'm fairly confident that there are indeed 6 attachments (or at least there were 6 attachments at some point), and I can see which are already available.
3. Entries that show just a blue "Download PDF" button (e.g. https://www.courtlistener.com/docket/60120427/us-dominion-inc-v-herring-networks-inc/?filed_after=&filed_before=&entry_gte=5&entry_lte=5&order_by=asc). Until yesterday, I assumed that meant that Courtlistener/RECAP believed that there was a document and that there were no attachments and that a free copy of the document was available.

But it turns out I was wrong about #3! It means that Courtlistener does not remember whether it even has information about the existence of attachments. In the ordinary course of things, the way that a "Download PDF" button shows up is for someone running the RECAP extension to buy the document on PACER, and if there are attachments and all the network requests work, then RECAP would upload the attachment page, if any, so there are probably no attachments.

But that probably is very fragile. And recap-fetch completely breaks it. To demonstrate, I submitted this request:

https://www.courtlistener.com/api/rest/v3/recap-fetch/?recap_document=244326103

and now this docket entry is a bit busted:

https://www.courtlistener.com/docket/60120427/us-dominion-inc-v-herring-networks-inc/?filed_after=&filed_before=&entry_gte=75&entry_lte=75&order_by=asc

It has a blue "Download PDF" button. It shows no attachments. It looks like this in the REST web UI (https://www.courtlistener.com/api/rest/v3/docket-entries/?docket__id=60120427&entry_number=75):

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "resource_uri": "https://www.courtlistener.com/api/rest/v3/docket-entries/238168376/",
            "id": 238168376,
            "docket": "https://www.courtlistener.com/api/rest/v3/dockets/60120427/",
            "recap_documents": [
                {
                    "resource_uri": "https://www.courtlistener.com/api/rest/v3/recap-documents/244326103/",
                    "id": 244326103,
                    "tags": [],
                    "absolute_url": "/docket/60120427/75/us-dominion-inc-v-herring-networks-inc/",
                    "date_created": "2023-03-14T12:18:37.898995-07:00",
                    "date_modified": "2023-05-04T13:48:43.737166-07:00",
                    "sha1": "49fa21bdf26b80b1a305bdc8b83b568885ca78e1",
                    "page_count": 2,
                    "file_size": 28708,
                    "filepath_local": "recap/gov.uscourts.dcd.234315/gov.uscourts.dcd.234315.75.0.pdf",
                    "filepath_ia": "",
                    "ia_upload_failure_count": null,
                    "thumbnail": null,
                    "thumbnail_status": 0,
                    "plain_text": "            Case 1:21-cv-02130-CJN Document 75 Filed 03/14/23 Page 1 of 2\n\n\n\n\n                      IN IN THE UNITED STATES DISTRICT COURT\n                           FOR THE DISTRICT OF COLUMBIA\n\n US DOMINION, INC., et al.,\n\n                      Plaintiffs/Counterclaim\n                      Defendants,                    No. 1:21-cv-02130-CJN\n        v.                                           Judge Carl J. Nichols\n HERRING NETWORKS, INC., et al.,\n\n                      Defendants/Counterclaim\n                      Plaintiffs/Third-Party\n                      Plaintiffs,\n\n        v.\n\n WILLIAM E. KENNARD, et al.,\n c/o AT&T Inc.\n 208 S. Akard St.\n Dallas, TX 75202,\n\n                      Third-Party Defendants.\n\n\n\n\n                        MOTION FOR ADMISSION PRO HAC VICE\n\n                Pursuant to Rule 83.2(c) of the Local Rules of the United States District Court for\n\nthe District of Columbia, undersigned counsel Amanda F. Davidoff, a member of the Bar of this\n\nCourt, respectfully moves that William B. Monahan be admitted pro hac vice in the above-\n\ncaptioned action.\n\n                In support of this motion, the undersigned represents:\n\n       1.        Mr. Monahan is a lawyer at Sullivan & Cromwell LLP, 125 Broad Street, New\n\nYork, New York 10004; telephone (212) 558-4000.\n\n       2.        As set forth in the Declaration of William B. Monahan accompanying this motion,\n\nMr. Monahan is a member in good standing of the State Bar of New York. Mr. Monahan is also\n\f            Case 1:21-cv-02130-CJN Document 75 Filed 03/14/23 Page 2 of 2\n\n\n\n\na member in good standing of the following federal bars: the Supreme Court of the United States,\n\nthe United States Courts of Appeals for the Second, Fourth, Fifth, Eighth, Ninth, and Tenth\n\nCircuits, the United States District Court for the Southern District of New York, the United States\n\nDistrict Court for the Eastern District of New York, the United States District Court for the\n\nNorthern District of New York, and the United States District Court for the Eastern District of\n\nMichigan.\n\n       3.        Mr. Monahan has not been admitted pro hac vice to practice before this Court\n\nwithin the last two years.\n\n                WHEREFORE, the undersigned moves this Court for an Order permitting\n\nMr. Monahan to appear pro hac vice in the above-captioned action as counsel for Third-Party\n\nDefendant William E. Kennard.\n\n\nDated: March 14, 2023\n\n\n                                          /s/ Amanda F. Davidoff\n                                          Amanda F. Davidoff, DC Bar No. 978033\n                                          [davidoffa@sullcrom.com\n](mailto:davidoffa@sullcrom.com\n)                                          SULLIVAN & CROMWELL LLP\n                                          1700 New York Avenue, N.W. Suite 700\n                                          Washington, D.C. 20006\n                                          Telephone:      (202) 956-7500\n\n                                          William B. Monahan (seeking pro hac vice admission)\n                                          [monahanw@sullcrom.com\n](mailto:monahanw@sullcrom.com\n)                                          SULLIVAN & CROMWELL LLP\n                                          125 Broad Street\n                                          New York, New York 10004\n                                          Telephone:       (212) 558-4000\n\n                                          Counsel for Third-Party Defendant\n                                          William E. Kennard\n\n\n\n\n                                                2\n\f",
                    "ocr_status": 2,
                    "date_upload": "2023-05-04T13:48:43.551735-07:00",
                    "document_number": "75",
                    "attachment_number": null,
                    "pacer_doc_id": "04509806653",
                    "is_available": true,
                    "is_free_on_pacer": null,
                    "is_sealed": null,
                    "document_type": 1,
                    "description": "Appear Pro Hac Vice"
                }
            ],
            "date_created": "2023-03-14T12:18:37.890434-07:00",
            "date_modified": "2023-04-20T06:48:04.065392-07:00",
            "date_filed": "2023-03-14",
            "time_filed": "14:45:48",
            "entry_number": 75,
            "recap_sequence_number": "2023-03-14.001",
            "pacer_sequence_number": 359,
            "description": "MOTION for Leave to Appear Pro Hac Vice :Attorney Name- William B. Monahan, Filing fee $ 100, receipt number ADCDC-9924762. Fee Status: Fee Paid. by William Kennard. (Attachments: # 1 Declaration of William B. Monahan, # 2 Certificate of Good Standing, # 3 Text of Proposed Order)(Davidoff, Amanda) (Entered: 03/14/2023)",
            "tags": []
        }
    ]
}

As far as I can tell, there is nothing that says "RECAP has no idea whether there are attachments". I would have expected a "attachment_page_updated_time":null or something roughly equivalent. And I would have expected something in the human-facing /docket/... page with a link that a PACER user can click to find out whether there are attachments.

(Yes, that docket entry has three attachments. I checked with RECAP uploads disabled.)

Am I making sense?

[mlissner]

My understanding is that the webpage/API don't always know whether there are attachments on a given entry, because to know that you have to go and try to download the attachment page (of which there are potentially hundreds of millions).

Are you saying that you somehow are able to remove attachment info from entries?

[amluto]

My understanding is that the webpage/API don't always know whether there are attachments on a given entry, because to know that you have to go and try to download the attachment page (of which there are potentially hundreds of millions).

Are you saying that you somehow are able to remove attachment info from entries?

Not really. What I'm saying is: the webpage/API doesn't know whether there are attachments, but it should know and it generally acts like it knows. And I'm able to confuse it.

When I navigate (manually) to PACER, for the main document in a docket entry, I get one of two things back: a Document Selection Menu showing attachments, their descriptions, and their page counts and prices. Or I get a Service Receipt (if I remember correctly -- PACER just decided to stop working for me while typing this email), which shows the number of pages and cost and lets me buy the document if I want.

I'm I'm running RECAP, it uploads the Document Selection Menu, but (AFAICT) it does not upload the Service Receipt. So, before I ever have the chance to manually upload a PDF, I (as a human being) know whether there are attachments. And if I do pay for and upload a PDF, if there were attachments, RECAP is likely to have received the Document Selection Menu and all is well.

But courtlistener doesn't know all this, because RECAP doesn't tell it, and RECAP doesn't know (I think) how to parse the Service Receipt. And it certainly reports errors when I send a request_type=3 request for a docket entry with no attachments.

Anyway, my suggestion would be to add some state (a new column, presumably) to the DocketEntry that tracks whether RECAP knows whether there are attachments. And teach RECAP to upload Service Receipts, and teach the backend to parse them and update the DocketEntry (and presumably also store the page count in the RecapDocument even if the user does not proceed to buy the document).

And then disallow request_type=2 requests for DocketEntries that don't know whether there are attachments, or alternatively to make the fetch logic automatically grab the menu (if any) if the DocketEntry does not know whether there are attachments.

Maybe I'm doing a better job explaining now?

[mlissner]

Yeah, I guess what I'm still missing is what problem does this solve? I guess you wish that you didn't have to make attachment page requests for pages where CL could know there weren't attachments?