llm4np: language modeling for neuropsychology reports

Introduction

We use the wizrd package to produce tooling for condensation and ontological tagging of neuropsychology reports.

Using gpt-4o-mini to condense an example report

For this code to run without modifications, environment variable OPENAI_API_KEY must have a valid setting.

library(wizrd)
library(llm4np)
pretty_rmd <- function(x) {
    x |> strwrap() |> paste(">", text = _, collapse = "\n") |> cat()
}

We have retrieved an example PDF of a neuropsychological (NP) evaluation from thepeerconsult.com.

Note: The parse_nppdf may emit warnings or errors during the table parsing phase. These refer to non-existent files, and seem to be false positives. They should be ignored.

parse_nppdf uses pdftools and tabulapdf to extract text and tables from reports.

nelsonasd = system.file("pdfs", "nelsonAutDeid.pdf", package="llm4np")
nels = parse_nppdf(nelsonasd)

## Error : '4.  Educators are also likely to be best able to support George’s emotional, behavioral, and social functioning' does not exist in current working directory ('/Users/vincentcarey/AUTISM/llm4np/vignettes').
## Error : '    ognitive Skills             ' does not exist in current working directory ('/Users/vincentcarey/AUTISM/llm4np/vignettes').

nels

## Parsed NP pdf with 14 pages and 5 tables.

We paste the report text into a single character vector for convenient processing.

nt = gsub("\\n", " ", paste(nels@text, collapse=" "))
substr(nt,1130,1210)

## [1] " is a highly gifted 12 year-old boy who is experiencing social, executive functio"

We also have a view of an autism ontology based on a 2015 paper by Alexa McCray and colleages.

data(nmdf)
head(nmdf)

##         tag                                           val
## 1 Class_634                 Obstetric Delivery Procedures
## 2 Class_385                             Perinatal History
## 3 Class_464 Attention Deficit Disorder with Hyperactivity
## 4 Class_455                              Mental Disorders
## 5 Class_689                   Psychiatric Hospitalization
## 6 Class_138                       Psychologic Indications

Now we create an agent that will be prompted to use the ontology in a summary of the NP evaluation.

ag = openai_agent("gpt-4o-mini") |> 
   instruct("You are a helpful assistant with training in neuropsychology.")
ag

## <wizrd::Agent> gpt-4o-mini
## @instructions: You are a helpful assistant with training in neuropsychology.
## @io: <wizrd::PlainTextFormat> → ← <wizrd::PlainTextFormat>
## @tools: 
## @params: 
## @backend: <wizrd::OpenAIAPIServer> https://api.openai.com

We now use a templated prompt, using the report text and the ontology in a summary.

prompted_agent <- ag |> prompt_as("Please summarize the text {txt} in five short paragraphs 
    of at most 45 words.  Use the concepts in the 'val' field of {df} as much as possible, 
    choosing the most specific term that is relevant, and when you do use these terms, please 
    append the associated value of the 'tag' field in parentheses, prefixing the tag with 
    'PTO:'.  After producing the paragraphs, please produce three lists of tags that were 
    used, corresponding to 'strengths', 'weaknesses' and 'needs'.  Do not use headings with 
    the lists, just provide the lists as sentences preceded by the category.") 
prompted_agent |> predict(list(txt=nt, df=nmdf)) |> pretty_rmd()

George Smith is a highly gifted 12-year-old diagnosed with Autism Spectrum Disorder, ADHD, and anxiety, exhibiting exceptional cognitive abilities, especially in verbal, nonverbal, and spatial problem-solving (PTO: “Cognitive Ability”). His performance aligns with that of a typical high school graduate, demanding tailored educational support (PTO: “Learning Disorders”).

Despite his strengths, George faces considerable challenges primarily in executive functioning, slow processing speed (PTO: “Attention Deficit Disorder with Hyperactivity”), and social communication, impacting peer interactions and emotional regulation (PTO: “Emotional Outbursts”). His rigidity and low tolerance for change complicate transitions and flexibility (PTO: “Reaction to Unexpected Change”).

Socially, George’s enthusiasm and unique interests may hinder relationship development, as he struggles with engagement and perspective-taking (PTO: “Recognition of Social Norms”). He is often unaware of social cues and tends to dominate conversations with his interests (PTO: “Immoderate Behavior”), increasing isolation and anxiety (PTO: “Social Anxiety”).

Ongoing support at home and school is crucial for nurturing George’s social skills, emotional wellbeing, and executive functioning (PTO: “Emotional Regulation and Control”). Strategies may include structured activities, specialized social skills training, and accommodations for processing speed within an Individualized Education Program (IEP) (PTO: “Primary Diagnosis”).

Overall, George has a compelling blend of strengths and vulnerabilities requiring integrated support to thrive academically and socially (PTO: “Social Competence”). Continuous assistance in providing structure and understanding his unique needs will ensure he can leverage his remarkable abilities effectively (PTO: “Planning”).

Strengths include: Cognitive Ability (PTO: “Cognitive Ability”), Learning Disorders (PTO: “Learning Disorders”), Emotional Regulation and Control (PTO: “Emotional Regulation and Control”), and Social Competence (PTO: “Social Competence”). Weaknesses include: Attention Deficit Disorder with Hyperactivity (PTO: “Attention Deficit Disorder with Hyperactivity”), Immoderate Behavior (PTO: “Immoderate Behavior”), Reaction to Unexpected Change (PTO: “Reaction to Unexpected Change”), and Emotional Outbursts (PTO: “Emotional Outbursts”). Needs include: Primary Diagnosis (PTO: “Primary Diagnosis”), Social Competence (PTO: “Social Competence”), and Planning (PTO: “Planning”).

A second example, no autism diagnosis present

sqspf = system.file("pdfs", "de-identified+report+1.pdf", package="llm4np")
sq = parse_nppdf(sqspf)
txt2 = gsub("\\n", " ", paste(sq@text, collapse=" "))  # concatenate text
prompted_agent |> predict(list(txt=txt2, df=nmdf)) |> pretty_rmd()

Joe is an 8-year-old boy with a history of premature birth, Tourette Syndrome, and Attention Deficit Hyperactivity Disorder (ADHD). Current assessments reveal concerns about his attentional control, impacting his academic performance. Joe has a positive family background but faces developmental delays.

Observations during the assessment showed that Joe appeared engaged, cooperative, and communicative, yet his attention was disrupted by verbal overflow. His cognitive assessment indicates borderline intellectual functioning, with attention, executive function, and motor control being significant areas of difficulty.

Language abilities were variable; spontaneous speech was adequate, but receptive and expressive skills showed some weaknesses due to attentional issues. Nonverbal processing and visual memory also presented challenges, particularly with tasks requiring complex integration.

Joe’s adaptive skills ratings are below age expectations, indicating difficulties in self-regulation, planning, and task organization. His medical history includes medication for anxiety and impulsivity, and he faces ongoing challenges related to his ADHD and Tourette syndrome, influencing his overall academic trajectory.

Recommendations suggest continuing small group instruction in reading and math, external support for task organization, and accommodations for testing due to his processing speed. Regular follow-ups with health and educational providers are essential for ongoing monitoring and support.

Strengths: Joe displays good verbal fluency (PTO: Ability to Convey Information), is cooperative during assessments (PTO: Personal Traits), and shows appropriate articulation (PTO: Expressive Language).

Weaknesses: He has attentional deficits (PTO: Attention and Focus), struggles with executive function (PTO: Executive Function), and has motor control difficulties (PTO: Gross Motor Skills).

Needs: Joe requires small group instruction in academics (PTO: Performance in School), needs behavioral support for planning (PTO: Planning), and benefits from structured routines at home (PTO: Home Life Skills).

Vincent J. Carey, stvjc at channing.harvard.edu

June 20, 2025

Introduction

Using gpt-4o-mini to condense an example report

A second example, no autism diagnosis present