Bug fix (Settings > Tall AI): a valid Gemini API key was reported as
refused, and a new user could not run TALL AI at all. Key validation sent a
real generateContent request to a hard-coded gemini-2.5-flash
(settings.R), and Google has stopped serving the 2.5 family to newly created
API keys — so the probe returned 404 ... no longer available to new users
and the app said "API key seems be not valid", whatever model was selected.
The first-run default was 2.5-flash too (loadGeminiModel()), so even a key
that got through had nothing it could call. Same defect as bibliometrix #637,
found by carrying that fix across.
.geminiAvailableModels()
asks for the model CATALOGUE (GET /v1beta/models) instead of invoking a
model, so no future retirement can break it. geminiValidateKey() returns
the key's verdict and the chosen model's availability as two separate
answers.grepl("HTTP\\s*[1-5][0-9]{2}") collapsed every
status from 100 to 599 into one "invalid key" message; 400/401 now mention
the key, 403 a permission, 404 the model, and a transport failure says it is
a connection problem. req_error() also lets Google's own message through:
resp_body_string() was being handed a hand-built list, so the real reason
was never shown.gemini_ai() was commented out, leaving status_code = NA, so
if (resp$status_code == 400) evaluated if (NA) and the user saw
"missing value where TRUE/FALSE needed" instead of "there is no network".gemini-3.5-flash-lite
(transport default, first-run default and model selector). A stored
2.5-flash — which was the old default rather than anyone's choice — is
carried forward; a deliberately chosen 2.5-flash-lite is kept. The 2.5
pair stays selectable under a Legacy group, since keys created before
the cut-off still reach it.3-flash-preview,
3.1-pro-preview): they carry their own expiry date. gemini-flash-latest
added.setGeminiAPI() removed: unreachable (no callers anywhere in the package)
and a duplicate of the same faulty probe. abstractive_summary(), also
unreachable and superseded by the inline block in documents.R, had its own
stale 2.0-flash default corrected and is now marked as superseded.Bug fix (Documents > Topic Modeling > Find optimal K): the page tuned LDA
with a different algorithm from the one that then estimates the model.
tmTuningAsync() fitted LDA(dtm, k, method = "VEM") while tmEstimate()
fits LDA(dtm, K, method = "Gibbs", control = list(iter = 500, seed = seed)),
so the number of topics a user was shown had been chosen for a model class
they never fitted — two different inference algorithms, one picking K for the
other. CTM was never affected (the same CTM() runs on both pages) and STM
tunes through stm::searchK, which fits stm models. The tuning now uses
method = "Gibbs" with tmEstimate()'s own control list, iterations
included, so the model that is scored and the model that is estimated are the
same fit. ⚠ The recommended K will change on existing corpora — that is
the point of the fix, not a side effect. evaluate_single_k() is kept in
step for the same reason, although nothing calls it.
Performance (Documents > Topic Modeling > Find optimal K): the grid is fitted
across cores instead of one model after another, and each worker now also
computes its own model's logLik and perplexity — which were half the wall
clock when they were left to the calling process — returning only the numbers
the metrics need (exp(beta) and len %*% gamma, a vector of length K)
rather than the fitted model. Measured on a 22,053-unit corpus over the
default grid K = 2..20: 36.7s to 5.8s, 6.4x. The metrics come back
bit-identical, which is what makes the change safe to describe as
performance: every fit is independent and carries the same seed, so the
result cannot depend on the order the workers finish in. If a cluster cannot
be created the grid is fitted serially rather than the page failing, and on a
grid small enough to finish instantly the cluster's own start-up dominates
(0.2s to 0.4s on a 593-unit corpus over K = 2..6). tmTuningAsync() no
longer returns the fitted models: nothing read them, and carrying them back
through the socket was the single largest cost.
Bug fix (Documents > Topic Modeling > Find optimal K): on the "Multi-Metric
Comparison" chart, two of the four curves were drawn upside down against the
chart's own y axis, which reads "Normalized Score (0 = worst, 1 = best)".
tmMultiMetricPlot() carried a per-metric flag named decreasing and its
normalisation branch treated that flag as "higher is better", while the flags
themselves had been assigned as though it meant "lower is better". Under LDA
and CTM this put the best K of CaoJuan2009 and of Perplexity — both
minimised — at the BOTTOM of a chart that says the bottom is worst; under STM
it did the same to Semantic Coherence and to the Lower Bound, both of
which are maximised. A reader comparing curves was therefore reading two of
them backwards. The flag is now higher_better, set from what each metric
actually wants, and 1 is the good end for every curve on both branches
(verified by execution: all eight metrics now plot their optimum at y = 1).
The recommended K values do not change, and could not have: find_elbow()
ignores its decreasing argument. Negating the metric reflects the whole
configuration about the x axis, and the distance from a point to a line is
invariant under reflection, so which.max(distances) returns the same index
either way — confirmed on 2000 random metric shapes, identical 2000/2000, and
on the patch itself, where every K came back unchanged. The argument is kept,
because it documents which way a metric runs, but it is now labelled as
having no effect so that nobody "fixes" the flags expecting a different K.
Documentation fix (Documents > Topic Modeling > Find optimal K): the "K Recommendation" tab described Deveaud 2014 as "Jensen-Shannon divergence between topic pairs. Lower = more separated topics." A divergence is larger when distributions are further apart, which is why Deveaud 2014 is maximised; the sentence said the opposite of both the statistic and the code beside it.
Bug fix (Documents > Topic Modeling > Model Estimation): the document labels
of theta were wrong under every estimation method, and the two branches were
wrong in two different ways. tmEstimate() (LDA and CTM) labelled each row
with unique(x$doc_id)[as.numeric(row.names(tmResult$topics))], but those row
names are topic_level_id values — the ANALYSIS UNIT, a sentence whenever
group includes sentence_id — not positions in the list of documents, so
indexing the document vector with a unit id returned NA for every unit past
the number of documents and misattributed the few that did resolve. Measured
on mobydick at sentence units (836 units, 10 documents): 826 of 836 labels
were NA, and of the 10 that were not, only ONE was right. stmEstimate()
took the first nrow(theta) entries of the unit list instead, which slides the
labels as soon as the unit ids are not the leading ones — they are neither
contiguous (LemmaSelection() drops units, so the ids skip: 3, 4, 5, 7, …) nor
guaranteed to start at 1 (empty units are dropped before the fit). Measured on
frankenstein at sentence units, 153 of 593 rows were attributed to the wrong
document — silently, since no label came back NA. The visible symptom of both
is "Topic by Docs Plot", whose y axis became a column of NA under LDA/CTM and
a plausible but wrong set of documents under STM. It went unnoticed because the
"Topics" selector defaults to Docs, where unit ids and document positions
coincide. Each row is now matched back to its own unit.
Bug fix (Import > Wikipedia pages): nothing on this path was URL-encoded, and
each of the three consequences was reachable from the ordinary use of the
menu. A page whose title contains a non-ASCII character — Zürich, Café,
Pokémon — made the API answer 400 Bad Request, and because one failing
page stops the extraction loop, a single such hit among the results left the
import with nothing. A search phrase containing & had everything after it
read as a new API parameter, so the query silently ran on the truncated
phrase. And a phrase containing an en dash was turned into one containing a
literal space by wikiSearch()'s own gsub("–", " ", …), which makes the URL
illegal and aborts the import with a connection error rather than the
"No results found!" message. Both the search phrase and the page title are now
passed through URLencode(reserved = TRUE).
Bug fix (Documents > Summarization > Extractive): textrankDocument() joined
the ranked sentences back to the corpus by the TEXT of the sentence
(by = c("sentence")), not by its id. Any document that repeats a sentence
therefore produced a many-to-many join: a text said k times contributed k^2
rows instead of k, and every copy inherited every paragraph that carries that
text. Because nrow(s) is what abstractingDocument() takes its slider
percentages of, the summary length moved with the duplication as well. On the
usairlines collection a four-sentence tweet whose text repeats three times
was rendered as ten sentences; mobydick_ita repeats 819 of its 12,356. The
join now uses the sentence id, as the neighbouring highlightSentences()
already did.
Bug fix (Documents > Summarization > Extractive): the view failed on a grouped
corpus, on its own default setting. "Summarize: Documents" lists original
document ids (ids() un-groups before listing), while the summary ran on
values$dfTag, still keyed by group — so the selected id was absent from the
data and the view stopped inside textrank_sentences() with
nrow(data) > 1 is not TRUE. Grouping under FEATURES and pressing Run was
enough to reach it. The frame is now un-grouped whenever the unit is not
"Groups".
Bug fix (Settings > Tall AI): the chosen Gemini model and output size were
never remembered. They were written to ~/.tall_gemini_model.txt and read
back from ~/tall/.tall_gemini_model.txt — a different file — so every
session started on 2.5-flash / medium whatever the user had picked. Two
further defects in the same path: the model observer saved
input$gemini_output_model, an input that does not exist (the control is
gemini_output_size), so choosing a model also dropped the output size; and
loadGeminiModel() tested length(model == 1) — the length of a logical
vector, hence always at least 1 — so it appended a third element to an
already complete pair. There is now a single geminiModelFile() used by both
the writer and the reader.
Bug fix (Settings > Working Folder): "Clean Model Cache" removed the whole
~/tall folder, not the model cache. It took the Gemini API key
(.tall_gemini_key.txt), the graph-export settings
(.tall_graph_settings.txt), the working-folder pointer (tallWD.tall) and
any .tall project the user had saved in that folder — none of which can be
downloaded again — under a button that names only the models. It now removes
~/tall/language_models and says so, and the alert states what has been kept.
Bug fix (start-up): an empty or blank ~/tall/tallWD.tall aborted the session
with argument is of length zero. readLines() of an empty file returns
character(0), and file.exists(character(0)) returns logical(0), which
if cannot evaluate — and wdFolder() is called from resetValues() before
there is any UI to report the error in. A blank pointer file is now treated as
no working folder, and removed.
Bug fix (Words > Word Embeddings > Similarity): the community-detection step
built its lookup table with data.frame(as.list(membership(cluster))), which
runs make.names() over the vertex names. Every term that is not a syntactic
R name therefore lost the join that follows and was silently dropped from
the network — no warning, no gap, just a smaller plot. This is not an exotic
case: it covers hyphens and apostrophes (e-mail), a leading digit (3d,
30), everything TALL's own entity tagging produces (hashtags, mentions,
emoji, URLs) and R's reserved words, so ordinary English terms such as next,
break, in and for disappeared too. On the US-airlines corpus 6,789 of
the 16,537 selected lemmas were affected, including 2 of the 100 words the
view is asked to plot. The table is now built directly from the membership
names and every node reaches the canvas.
Bug fix (Words > Word Embeddings > Similarity): when no pair of words reached
the hard-coded 0.5 similarity cutoff the empty edge list reached
graph_from_data_frame(), and the view died with Can't rename columns that don't exist. Column 'V1' doesn't exist. — a dplyr message with nothing to
connect it to the cutoff. The network now renders a single label saying that
no pair reaches the threshold.
Bug fix (Words > Word Embeddings > Similarity > UMAP): with a single word to
place, the label de-overlap routine ran 1:(nrow(df) - 1), which counts
down to 0, so it compared row 0 with row 1 and stopped on missing value where TRUE/FALSE needed. Nothing can overlap with itself, and the loop is now
skipped in that case.
Bug fix (Words > Word Embeddings): word2vec keeps a </s> sentence-boundary
sentinel in its model matrix. It is not a word of the corpus and its vector is
essentially untrained, but every consumer read the matrix with as.matrix()
and included it: on Frankenstein its norm is 4.47 against a mean absolute
component of 0.81 for real terms, which made it the Min or the Max in 10 of
the 20 dimensions of the Training tab, shifted Kurtosis by up to 183 and
dropped the variance explained by PC1 from 0.591 to 0.501 — in the very chart
that tab exists to show. It could also surface as a neighbour in the
similarity network and as a row of the exported matrix. A new w2vMatrix()
accessor drops it, and all consumers go through it.
Bug fix (Documents > Supervised Classification > Download Results): the
export read test_data$target, but the model frame names that column
.target_class, and "target" is not a prefix of it, so $ could not reach it
either. Two outcomes, both reproduced on a real corpus: normally $target
returned NULL and tibble() raised a recycling error inside the download
handler, so the whole .xlsx failed and none of its five sheets was written;
and on a corpus that happens to contain a term whose sanitised column name is
exactly target — the US-airlines tweets do — $ resolved to that TF-IDF
column instead, and the file was written with numbers in the Actual column
and Correct false on every row. The lookup is now [[".target_class"]],
which also cannot fall back to partial matching.
Bug fix (Documents > Supervised Classification): ranger's
prediction.error was displayed as "OOB Prediction Error (%)" in the training
summary, the success dialog and the exported workbook. For a probability = TRUE forest that field is the out-of-bag Brier score, not a
misclassification rate — 0.21 on the reference run, which invited reading it
as "21% of documents misclassified". It is now labelled and formatted as a
Brier score.
Bug fix (Documents > Supervised Classification): the training-set accuracy was computed from in-sample predictions, so it is optimistically biased by construction (0.92 against 0.75 on the held-out set in the reference run) and read like a second, reassuring validation score. It is now labelled "Train Set Accuracy (in-sample)".
Bug fix (Documents > Supervised Classification): training word embeddings from
this menu overwrote the shared values$w2v_model used by Words > Embeddings
with a differently parameterised, stopword-free model, while leaving
values$w2v_stats and values$df_EmbeddingDims describing the previous one —
so the embedding views silently changed under the user. The model trained here
is now kept inside the classification menu and the shared one is only read.
Bug fix (Documents > Supervised Classification): a failed training run left
the results panel showing the previous model, because the error handler did
not clear the trained flag.
Bug fix (Documents > Supervised Classification): the prerequisite check
aborted with "missing value where TRUE/FALSE needed" when docSelected
contained any NA, since sum() then returns NA and if (NA) is an error.
Bug fix (Features > Feature Roles): applying a keyness role to a binary variable that contains missing values aborted with "the condition has length
1", after
keyness_grouphad already been written to the corpus — so the roles were applied but the confirmation never appeared. The per-group document count did not drop theNAgroup, sodoc_groups$n[doc_groups$keyness_group == 1]returned two elements, the success message became a length-2 vector and the finalif (success_message == "")threw. The count now filters the missing group and guards an empty one, as the multi-category branch already did.
Bug fix (Features > Feature Roles): the time and label roles were assigned
before the keyness groups were validated, and the validation aborts with
return(). Rejecting an Apply for missing or overlapping group assignments
therefore left a half-applied state (time and label silently assigned, keyness
not). The group checks now run first, so a rejected Apply changes nothing.
Bug fix (Features > Feature Roles): the category list of the keyness variable was read from the raw token-level column, while the Group 1 / Group 2 pickers offer the categories of the selected documents. With a filter active the two could disagree — a variable reduced to two categories by the filter still took the "more than 2 categories" branch and demanded an assignment the pickers could not offer. Both now use the same document-level, filtered set.
Bug fix (Features > Feature Roles > Preview): the "Documents per period" table
of a date variable was ordered by ascending document count (sort() on a
table() sorts the counts), not chronologically.
Bug fix (Features > Feature Roles): aggregating a date-time variable by Day
did not aggregate at all — as.character() on a POSIXct keeps the clock
time, so every distinct second became its own period. It now formats to the
calendar day.
Bug fix (Features): noGroupLabels() did not reserve time_agg, the column
the time role derives, although it reserves keyness_group. After assigning a
date time role, tall's own aggregation key appeared as a user feature in
Filters, Groups (where grouping by it would re-key the documents), the topic
model covariates and Feature Roles itself, and its presence alone could unlock
the FEATURES menu.
Bug fix (Pre-processing > Multi-Word Creation / Multi-Word by a List): the
words absorbed into a multi-word were left in the corpus as separate tokens.
Merging "natural philosophy" produced the multi-word AND kept philosophy as
its own selected NOUN, so every downstream count (vocabulary, keyness,
networks, topic models) saw the constituents twice. Two chained causes, both
fixed: (1) the vectorised rewrite of the tagging step nested what used to be
two sequential ifelse() calls, and on an absorbed position multiword is
NA, so the outer test evaluated to NA and the NGRAM_MERGED branch became
unreachable — the tags now come from an explicit "was absorbed" flag, which
also leaves a genuinely NA term alone; (2) applyRake() kept a
NGRAM_MERGED row only when its own multiword was among the selected
keywords, but that value is NA on those rows by construction, so they were
always dropped and the join restored the original tags. The tagging step now
reports which keyword absorbed each position (mw_owner) and applyRake()
matches the constituents through it, so a partial selection keeps exactly the
constituents of the keywords you ticked. Constituents are now tagged
NGRAM_MERGED with POSSelected = FALSE — their text stays in the sentence,
they are only excluded from the analyses — and rakeReset() ("Back") restores
the previous selection from POSSelected_original_nomultiwords.
Bug fix (KWIC > In-Document Plot > View): clicking the "View" button of a
document whose annotation contains an unresolved lemma (NA) aborted the
document modal with "missing value where TRUE/FALSE needed". Comparing an NA
term with the query yields NA rather than FALSE, and buildDocumentHTML()
branches on that value once per token. The match flag now treats an
unresolvable term as a non-match, and the token branch uses isTRUE() so the
helper stays total for any caller. Documents without NA terms render exactly
as before, byte for byte. Note the table lists every selected document,
including those whose frequency is NA, so the button was reachable on any
corpus with an unresolved lemma (e.g. a single upos = "X" token).
Bug fix (Overview > Morphological Features): parseMorphFeatures() built its
match mask from the vector returned by regmatches(), which is compacted to
the matching elements only. Being all-TRUE and shorter than the input, the mask
was recycled when used to index the full-length result, so the extracted values
were sprayed cyclically across every token instead of landing on the tokens
that actually carry the feature. As a consequence no row was dropped as NA:
the feature distribution bar chart reported counts inflated to the whole
corpus token count (percentages were approximately right, absolute counts were
not) and the feature x part-of-speech cross-tabulation was meaningless. The
mask is now derived from regexpr() over the full column, so non-matching
tokens correctly stay NA.
Bug fix (Reinert clustering): in the greedy reallocation step (switch_docs),
the document to move was looked up in the original CA ordering instead of the
current group order. After the first switch this could move the wrong segment
and made the clustering depend on the arbitrary sign of the SVD first axis
(i.e., results could differ across platforms/LAPACK builds on the same data).
Partitions computed with previous versions may change slightly.
Changelog:
New Analyses
SVO Triplets (Subject-Verb-Object)
Syntactic Complexity
Emotion Analysis (NRC EmoLex)
Noun Phrase Extraction
Topic Modeling (major enhancement)
Image Export System (complete rewrite)
Homepage & References
Settings & UI
Dependency Parsing Features
Architectural Refactoring
Code Quality
Bug Fixes
New Features & UI/UX Improvements
Bug Fixes
New Features & Modules -Supervised Classification -New Module: Implemented a full Supervised Classification interface within the "Documents" section. - Workflow: Added support for two analysis approaches: DTM (Document-Term Matrix) and Embeddings.
Performance & Code Refactoring
UI/UX & Visualization Improvements
Visualizations
Bug Fixes & Logic Updates