Latin words by regex; CJK counted per character when min length is 1.
Latin tokens from [a-z0-9]+; CJK per character when min length is 1.
| Token | Count |
|---|---|
| the | 2 |
| quick | 1 |
| brown | 1 |
| fox | 1 |
| jumps | 1 |
| over | 1 |
| lazy | 1 |
| dog | 1 |
This is not NLP segmentation for Chinese—counts are indicative only.