multigrid.ai

Online Evaluation: Grading Production Traffic

The article explains how online evaluation (grading production traffic) complements offline evaluation by detecting input drift, provider-side changes, long-tail failures, and unexpected real inputs. It recommends a two-layer approach: inexpensive, programmatic guardrails applied to 100% of responses for enforcement, and judge-graded quality metrics on a sampled, asynchronous basis for measurement. Sampling should be determined by statistical precision needs (example: 1,225 graded requests/week for ±2 percentage points at 95% confidence) rather than an arbitrary percent of traffic. The piece describes weighted sampling using inclusion probabilities (Horvitz-Thompson/Hajek estimators) to oversample suspicious strata without biasing population estimates, and operational best practices for where and how grading should run and how to handle data/privacy constraints.

Abjad Scripts Leave Vowels Ambiguous for AI

The article explains how abjad writing systems like Arabic and Hebrew typically record consonants but omit short vowels, creating one-to-many mappings between written forms and spoken words. Because most training corpora for language models and NLP systems are unvocalised, models must guess vowel patterns when required to produce vocalised output. Models rely on syntactic position, collocation, corpus frequency, and dialect to disambiguate. This ambiguity causes practical failures in tasks that require explicit vowels — notably TTS, transliteration, exact matching/deduplication, search, and OCR. Recommended handling includes normalising text for indexing (folding alef variants, removing harakat/tatweel/niqqud), treating diacritisation as an explicit uncertain step, supplying as much context as possible, and keeping the original stored form alongside any derived vocalised form.

Ontologies, Taxonomies, and Schemas Explained

This technical article explains the differences between controlled vocabularies, taxonomies, and ontologies for knowledge graphs. A controlled vocabulary is a closed list of defined terms; a taxonomy arranges those terms with broader/narrower relations (SKOS) to enable roll-up queries; and an ontology (RDFS/OWL) adds axioms and property semantics so a reasoner can derive facts. The author warns that ontologies are costlier to build and maintain, that RDFS/OWL operate under an open-world assumption (so domain/range declarations infer types rather than validate), and that validation (e.g., SHACL) is usually what teams want to prevent bad data. Recommended pragmatic practice: start with vocabularies and taxonomies, add minimal ontology features where they materially save work, and reserve full reasoning for interchange scenarios.