LLM Integration

LLM extraction is optional — macrox works fully offline without it. Enable it when positional methods fail: multi-level headers, irregular bounding boxes, or scanned PDFs where Docling is not available.

Everything here requires the ellmer package and at least one provider API key.

Provider setup

Install ellmer once:

install.packages("ellmer")

Set your API key as an environment variable. The easiest place is .Renviron (one key per provider — add only the one you use):

ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
GEMINI_API_KEY=AI...

Edit .Renviron from R:

usethis::edit_r_environ()   # requires usethis; restart R after saving

Setting global defaults

Add these lines to .Rprofile so you never have to specify provider or model on each call:

options(
  macrox.llm.provider = "anthropic",
  macrox.llm.model    = "claude-opus-4-8"
)

Supported providers: "anthropic", "openai", "google_gemini", "openrouter", "groq", "ollama", "openai_compatible", and any other provider for which ellmer exports a chat_<provider>() constructor.

Session-level configuration

Call mx_configure_llm() once per session to avoid repeating provider/model on every extraction call:

library(macrox)

sess <- mx_session("report.pdf")

# Option 1: provider + model (uses global options as defaults)
sess |> mx_configure_llm(provider = "anthropic", model = "claude-opus-4-8")

# Option 2: pass a fully-configured ellmer Chat object for maximum control
chat <- ellmer::chat_anthropic(
  model  = "claude-opus-4-8",
  system = "Extract data exactly as shown in the document."
)
sess |> mx_configure_llm(chat = chat)

Once configured, all subsequent select_table_llm() and select_item() calls on the session use that provider without further arguments.

Local models via Ollama

Run models locally — no API key needed:

sess |> mx_configure_llm(
  provider = "ollama",
  model    = "llama3.2-vision:11b"   # any Ollama vision model
)

OpenAI-compatible endpoints

Works with LM Studio, vLLM, and any server that speaks the OpenAI API:

sess |> mx_configure_llm(
  provider = "openai_compatible",
  model    = "mistral-nemo",
  base_url = "http://localhost:1234/v1"
)

Extracting tables with select_table_llm()

Two extraction modes:

Mode	When to use
`"structured"` (default)	You know the columns; supply a `schema` for typed output
`"page"`	Unknown layout; LLM converts the full page to markdown

Structured mode

area <- sess |> locate_area(page = 5)

sess |> select_table_llm("livestock",
  page   = 5,
  area   = area,
  schema = c(
    Breed   = "character",
    Male    = "integer",
    Female  = "integer",
    Total   = "integer"
  )
)

schema is a named character vector: names become column names, values are R types ("character", "integer", "numeric"). The LLM receives the column names as guidance; macrox converts the returned strings to the declared types.

Page mode

# No area or schema needed — the LLM finds and parses all tables
sess |> select_table_llm("summary",
  page = 3,
  mode = "page"
)

# Multiple tables on one page — pick by index
sess |> select_table_llm("appendix",
  page        = 3,
  mode        = "page",
  table_index = 2
)

Extra instructions

Use prompt to add free-text guidance after the base prompt:

sess |> select_table_llm("financials",
  page   = 12,
  schema = c(Quarter = "character", Revenue = "numeric", Cost = "numeric"),
  prompt = "Ignore the 'YTD' row at the bottom. Revenue and Cost are in thousands."
)

Multi-level headers

Government tables often have two header rows. header_rows = 2 tells the LLM to flatten them with _ separators:

sess |> select_table_llm("breed_sire",
  page        = 16,
  area        = area,
  header_rows = 2,
  schema      = c(
    Group_A.Jan = "integer",
    Group_A.Feb = "integer",
    Group_B.Sub1 = "integer"
  )
)

Updating a schema after extraction

If the auto-detected columns need adjusting, use update_llm_schema() to revise without re-browsing:

# After initial extraction, refine column names and types
sess |> update_llm_schema("livestock",
  schema = c(
    Breed    = "character",
    Bulls    = "integer",
    Cows     = "integer",
    Heifers  = "integer",
    Total    = "integer"
  )
)

# Revise the prompt too
sess |> update_llm_schema("livestock",
  schema = c(Breed = "character", Total = "integer"),
  prompt = "Only extract the Breed and Total columns."
)

Extracting metadata fields with select_item()

select_item() extracts a single value that sits outside any table — a publication date, an invoice number, a report title. Results accumulate in sess$items alongside sess$tables.

# Extract from all-pages text (no page rendering needed)
sess |> select_item("pub_date",
  prompt = "Publication or report date in the header or footer."
)

# Cast the result directly
sess |> select_item("pub_date",
  prompt = "Publication date.",
  cast   = "date:%d/%m/%Y"
)

# Extract from a specific page image for better accuracy
sess |> select_item("issue_number",
  prompt = "Issue or reference number shown on the cover page.",
  page   = 1L
)

# View all extracted items
sess |> show_items()

Supported cast values: "character" (default), "numeric", "integer", "date:<strptime_format>".

Revising an item extraction

sess |> update_item("pub_date",
  prompt = "Date shown at the top of page 1 in DD/MM/YYYY format.",
  cast   = "date:%d/%m/%Y"
)

GLiNER2 — local metadata extraction

GLiNER2 is a local NLP model (~205 MB) that extracts named entities from text without any API key. It runs fully offline after a one-time model download.

Setup

setup_gliner()   # downloads the model into a managed virtualenv (~500 MB)
# Restart R before first use

Single fields

sess |> select_item("invoice_number",
  prompt  = "Invoice ID or reference number",
  backend = "gliner"
)

sess |> select_item("total_amount",
  prompt  = "Grand total amount payable",
  page    = 1L,
  cast    = "numeric",
  backend = "gliner"
)

# Retrieve all occurrences (e.g. all dates mentioned)
sess |> select_item("dates_mentioned",
  prompt      = "Date or dates mentioned in the document",
  backend     = "gliner",
  all_matches = TRUE
)

Batch extraction (faster for many fields)

GLiNER runs one inference pass for all fields — much faster than calling select_item() in a loop:

sess |> select_items_batch(
  items = c(
    invoice_number = "Invoice ID or reference number",
    vendor_name    = "Supplier or vendor name",
    total_amount   = "Grand total amount payable",
    due_date       = "Payment due date"
  ),
  cast = c(
    total_amount = "numeric",
    due_date     = "date:%d/%m/%Y"
  )
)

Structured records with select_struct()

select_struct() uses GLiNER2’s extract_json to pull multiple records with a defined schema — useful for line-item tables that GLiNER handles better than tabulapdf:

sess |> select_struct("line_items",
  fields = c(
    description = "Item description or product name",
    qty         = "Quantity ordered",
    unit_price  = "Unit price per item",
    total       = "Line total amount"
  ),
  list_fields  = character(0),             # none in this case
  page         = 2L,
  gliner_model = "fastino/gliner2-base-v1" # or "-large-v1" for better accuracy
)

# Convert raw records to a data frame (includes per-field confidence scores)
sess |> struct_to_df("line_items",
  min_confidence = 0.6   # drop low-confidence records; NULL keeps all
)

# Now available as sess$tables[["line_items"]] for downstream steps

The resulting table has value columns plus <field>_conf columns for per-field GLiNER confidence scores. Use min_confidence to filter out uncertain extractions.

Multi-page tables with LLM

When a table spans several pages and has an irregular or multi-level header, combine stack_pages() with method = "llm":

sess |> stack_pages("breed_all",
  pages  = 10:14,
  method = "llm",
  schema = c(
    Breed  = "character",
    Month  = "character",
    Male   = "integer",
    Female = "integer"
  )
)

stack_pages() extracts from each page independently and row-binds the results. The header_match = TRUE default discards repeated header rows on pages 2+.