## Tokenizer class [​](/content/docs/sdk/core.tokenizer#tokenizer-class "Direct link to Tokenizer class"/index.html)

**Signature:**

```typescript
export declare class Tokenizer
```

## Constructors [​](/content/docs/sdk/core.tokenizer#constructors "Direct link to Constructors"/index.html)

| Constructor | Modifiers | Description |
| --- | --- | --- |
| [(constructor)(str, keywords, operators, options)](/content/docs/sdk/core.tokenizer._constructor_/index.html) |  | Constructs a new instance of the `Tokenizer` class |

## Methods [​](/content/docs/sdk/core.tokenizer#methods "Direct link to Methods"/index.html)

| Method | Modifiers | Description |
| --- | --- | --- |
| [tokenize()](/content/docs/sdk/core.tokenizer.tokenize) |  |  |

- [Tokenizer class](/content/docs/sdk/core.tokenizer#tokenizer-class/index.html)
- [Constructors](/content/docs/sdk/core.tokenizer#constructors/index.html)
- [Methods](/content/docs/sdk/core.tokenizer#methods/index.html)