📚 CBSE Grade-10 Study Guide Open in the Study Guide single page app →
HomeArtificial Intelligence

Artificial Intelligence — CBSE Class 10 board question

Q1. [1]
______ is a term used for any word or number or special character occurring in a sentence. (Token / Punctuator)
Previously asked in CBSE board exam
2023 104 Q5 (i)
Generated by claude-sonnet-4-6 · 2026-06-21 03:18 · grounding rag
Model Answer

Token is a term used for any word or number or special character occurring in a sentence.

Source: Chapter 6, Section 6.3 / Text Normalisation – Tokenisation

Explanation

The examiner expects the single correct term "Token." In tokenisation (a step of Text Normalisation in NLP), the text is broken into smaller units called tokens — these can be words, numbers, or special characters. Do not write "Punctuator," as that refers only to punctuation marks, not all types of units in a sentence.

If a question refers to an image, map, graph or diagram that is not shown here, open the Study Guide single page app, go to Library and find the actual CBSE question paper. The original papers are also available on the CBSE website: cbse.gov.in.
Previous-year CBSE Grade 10 board exam questions, organised by subject and chapter, each with a model answer — free to read and print.