Skip to content
Snippets Groups Projects
Verified Commit 1ee0e0f0 authored by Volker Schukai's avatar Volker Schukai :alien:
Browse files

chore: vendoring

parent b6c0bf23
Branches
Tags
No related merge requests found
Showing
with 3568 additions and 0 deletions
# Tokenizer
[![Build Status](https://github.com/bzick/tokenizer/actions/workflows/tokenizer.yml/badge.svg)](https://github.com/bzick/tokenizer/actions/workflows/tokenizer.yml)
[![codecov](https://codecov.io/gh/bzick/tokenizer/branch/master/graph/badge.svg?token=MFY5NWATGC)](https://codecov.io/gh/bzick/tokenizer)
[![Go Report Card](https://goreportcard.com/badge/github.com/bzick/tokenizer?rnd=2)](https://goreportcard.com/report/github.com/bzick/tokenizer)
[![GoDoc](https://godoc.org/github.com/bzick/tokenizer?status.svg)](https://godoc.org/github.com/bzick/tokenizer)
Tokenizer — parse any string, slice or infinite buffer to any tokens.
Main features:
* High performance.
* No regexp.
* Provides [simple API](https://pkg.go.dev/github.com/bzick/tokenizer).
* Supports [integer](#integer-number) and [float](#float-number) numbers.
* Supports [quoted string or other "framed"](#framed-string) strings.
* Supports [injection](#injection-in-framed-string) in quoted or "framed" strings.
* Supports unicode.
* [Customization of tokens](#user-defined-tokens).
* Autodetect white space symbols.
* Parse any data syntax (xml, [json](https://github.com/bzick/tokenizer/blob/master/example_test.go), yaml), any programming language.
* Single pass through the data.
* Parses [infinite incoming data](#parse-buffer) and don't panic.
Use cases:
- Parsing html, xml, [json](./example_test.go), yaml and other text formats.
- Parsing huge or infinite texts.
- Parsing any programming languages.
- Parsing templates.
- Parsing formulas.
For example, parsing SQL `WHERE` condition `user_id = 119 and modified > "2020-01-01 00:00:00" or amount >= 122.34`:
```go
// define custom tokens keys
const (
TEquality = 1
TDot = 2
TMath = 3
)
// configure tokenizer
parser := tokenizer.New()
parser.DefineTokens(TEquality, []string{"<", "<=", "==", ">=", ">", "!="})
parser.DefineTokens(TDot, []string{"."})
parser.DefineTokens(TMath, []string{"+", "-", "/", "*", "%"})
parser.DefineStringToken(`"`, `"`).SetEscapeSymbol(tokenizer.BackSlash)
// create tokens stream
stream := parser.ParseString(`user_id = 119 and modified > "2020-01-01 00:00:00" or amount >= 122.34`)
defer stream.Close()
// iterate over each token
for stream.Valid() {
if stream.CurrentToken().Is(tokenizer.TokenKeyword) {
field := stream.CurrentToken().ValueString()
// ...
}
stream.Next()
}
```
tokens stram:
```
string: user_id = 119 and modified > "2020-01-01 00:00:00" or amount >= 122.34
tokens: |user_id| =| 119| and| modified| >| "2020-01-01 00:00:00"| or| amount| >=| 122.34|
| 0 | 1| 2 | 3 | 4 | 5| 6 | 7 | 8 | 9 | 10 |
0: {key: TokenKeyword, value: "user_id"} token.Value() == "user_id"
1: {key: TEquality, value: "="} token.Value() == "="
2: {key: TokenInteger, value: "119"} token.ValueInt() == 119
3: {key: TokenKeyword, value: "and"} token.Value() == "and"
4: {key: TokenKeyword, value: "modified"} token.Value() == "modified"
5: {key: TEquality, value: ">"} token.Value() == ">"
6: {key: TokenString, value: "\"2020-01-01 00:00:00\""} token.ValueUnescaped() == "2020-01-01 00:00:00"
7: {key: TokenKeyword, value: "or"} token.Value() == "and"
8: {key: TokenKeyword, value: "amount"} token.Value() == "amount"
9: {key: TEquality, value: ">="} token.Value() == ">="
10: {key: TokenFloat, value: "122.34"} token.ValueFloat() == 122.34
```
More examples:
- [JSON parser](./example_test.go)
## Begin
### Create and parse
```go
import (
"github.com/bzick/tokenizer"
)
var parser := tokenizer.New()
parser.AllowKeywordUnderscore() // ... and other configuration code
```
There is two ways to **parse string or slice**:
- `parser.ParseString(str)`
- `parser.ParseBytes(slice)`
The package allows to **parse an endless stream** of data into tokens.
For parsing, you need to pass `io.Reader`, from which data will be read (chunk-by-chunk):
```go
fp, err := os.Open("data.json") // huge JSON file
// check fs, configure tokenizer ...
stream := parser.ParseStream(fp, 4096).SetHistorySize(10)
defer stream.Close()
for stream.IsValid() {
// ...
stream.Next()
}
```
## Embedded tokens
- `tokenizer.TokenUnknown` — unspecified token key.
- `tokenizer.TokenKeyword` — keyword, any combination of letters, including unicode letters.
- `tokenizer.TokenInteger` — integer value
- `tokenizer.TokenFloat` — float/double value
- `tokenizer.TokenString` — quoted string
- `tokenizer.TokenStringFragment` — fragment framed (quoted) string
### Unknown token — `tokenizer.TokenUnknown`
A token marks as `TokenUnknown` if the parser detects an unknown token:
```go
parser.ParseString(`one!`)
```
```
{
{
Key: tokenizer.TokenKeyword
Value: "One"
},
{
Key: tokenizer.TokenUnknown
Value: "!"
}
}
```
By default, `TokenUnknown` tokens are added to the stream.
To exclude them from the stream, use the `tokenizer.StopOnUndefinedToken()` method
```
{
{
Key: tokenizer.TokenKeyword
Value: "one"
}
}
```
Please note that if the `tokenizer.StopOnUndefinedToken` setting is enabled, then the string may not be fully parsed.
To find out that the string was not fully parsed, check the length of the parsed string `stream.GetParsedLength()`
and the length of the original string.
### Keywords
Any word that is not a custom token is stored in a single token as `tokenizer.TokenKeyword`.
The word can contains unicode characters, numbers (see `tokenizer.AllowNumbersInKeyword ()`) and underscore (see `tokenizer.AllowKeywordUnderscore ()`).
```go
parser.ParseString(`one two четыре`)
```
```
tokens: {
{
Key: tokenizer.TokenKeyword
Value: "one"
},
{
Key: tokenizer.TokenKeyword
Value: "two"
},
{
Key: tokenizer.TokenKeyword
Value: "четыре"
}
}
```
### Integer number
Any integer is stored as one token with key `tokenizer.Token Integer`.
```go
parser.ParseString(`223 999`)
```
```
tokens: {
{
Key: tokenizer.TokenInteger
Value: "223"
},
{
Key: tokenizer.TokenInteger
Value: "999"
},
}
```
To get int64 from the token value use `stream.GetInt()`:
```go
stream := tokenizer.ParseString("123")
fmt.Print("Token is %d", stream.CurrentToken().GetInt()) // Token is 123
```
### Float number
Any float number is stored as one token with key `tokenizer.TokenFloat`. Float number may
- have point, for example `1.2`
- have exponent, for example `1e6`
- have lower `e` or upper `E` letter in the exponent, for example `1E6`, `1e6`
- have sign in the exponent, for example `1e-6`, `1e6`, `1e+6`
```
tokenizer.ParseString(`1.3e-8`):
{
{
Key: tokenizer.TokenFloat
Value: "1.3e-8"
},
}
```
To get float64 from the token value use `token.GetFloat()`:
```go
stream := tokenizer.ParseString("1.3e2")
fmt.Print("Token is %d", stream.CurrentToken().GetFloat()) // Token is 130
```
### Framed string
Strings that are framed with tokens are called framed strings. An obvious example is quoted a string like `"one two"`.
There quotes — edge tokens.
You can create and customize framed string through `tokenizer.AddQuote()`:
```go
const TokenDoubleQuotedString = 10
tokenizer.DefineStringToken(TokenDoubleQuotedString, `"`, `"`).SetEscapeSymbol('\\')
stream := tokenizer.ParseString(`"two \"three"`)
```
```
{
{
Key: tokenizer.TokenString
Value: "\"two \\"three\""
},
}
```
To get a framed string without edge tokens and special characters, use the `stream.ValueUnescape()` method:
```go
v := stream.CurrentToken().ValueUnescape() // result: two "three
```
The method `token.StringKey()` will be return token string key defined in the `DefineStringToken`:
```go
stream.CurrentToken().StringKey() == TokenDoubleQuotedString // true
```
### Injection in framed string
Strings can contain expression substitutions that can be parsed into tokens. For example `"one {{two}} three"`.
Fragments of strings before, between and after substitutions will be stored in tokens as `tokenizer.TokenStringFragment`.
```go
const (
TokenOpenInjection = 1
TokenCloseInjection = 2
TokenQuotedString = 3
)
parser := tokenizer.New()
parser.DefineTokens(TokenOpenInjection, []string{"{{"})
parser.DefineTokens(TokenCloseInjection, []string{"}}"})
parser.DefineStringToken(TokenQuotedString, `"`, `"`).AddInjection(TokenOpenInjection, TokenCloseInjection)
parser.ParseString(`"one {{ two }} three"`)
```
Tokens:
```
{
{
Key: tokenizer.TokenStringFragment,
Value: "one"
},
{
Key: TokenOpenInjection,
Value: "{{"
},
{
Key: tokenizer.TokenKeyword,
Value: "two"
},
{
Key: TokenCloseInjection,
Value: "}}"
},
{
Key: tokenizer.TokenStringFragment,
Value: "three"
},
}
```
Use cases:
- parse templates
- parse placeholders
## User defined tokens
The new token can be defined via the `DefineTokens` method:
```go
const (
TokenCurlyOpen = 1
TokenCurlyClose = 2
TokenSquareOpen = 3
TokenSquareClose = 4
TokenColon = 5
TokenComma = 6
TokenDoubleQuoted = 7
)
// json parser
parser := tokenizer.New()
parser.
DefineTokens(TokenCurlyOpen, []string{"{"}).
DefineTokens(TokenCurlyClose, []string{"}"}).
DefineTokens(TokenSquareOpen, []string{"["}).
DefineTokens(TokenSquareClose, []string{"]"}).
DefineTokens(TokenColon, []string{":"}).
DefineTokens(TokenComma, []string{","}).
DefineStringToken(TokenDoubleQuoted, `"`, `"`).SetSpecialSymbols(tokenizer.DefaultStringEscapes)
stream := parser.ParseString(`{"key": [1]}`)
```
## Known issues
* zero-byte `\0` ignores in the source string.
## Benchmark
Parse string/bytes
```
pkg: tokenizer
cpu: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
BenchmarkParseBytes
stream_test.go:251: Speed: 70 bytes string with 19.689µs: 3555284 byte/sec
stream_test.go:251: Speed: 7000 bytes string with 848.163µs: 8253130 byte/sec
stream_test.go:251: Speed: 700000 bytes string with 75.685945ms: 9248744 byte/sec
stream_test.go:251: Speed: 11093670 bytes string with 1.16611538s: 9513355 byte/sec
BenchmarkParseBytes-8 158481 7358 ns/op
```
Parse infinite stream
```
pkg: tokenizer
cpu: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
BenchmarkParseInfStream
stream_test.go:226: Speed: 70 bytes at 33.826µs: 2069414 byte/sec
stream_test.go:226: Speed: 7000 bytes at 627.357µs: 11157921 byte/sec
stream_test.go:226: Speed: 700000 bytes at 27.675799ms: 25292856 byte/sec
stream_test.go:226: Speed: 30316440 bytes at 1.18061702s: 25678471 byte/sec
BenchmarkParseInfStream-8 433092 2726 ns/op
PASS
```
package tokenizer
import (
"strconv"
"strings"
)
// Stream iterator via parsed tokens.
// If data reads from an infinite buffer then the iterator will be read data from reader chunk-by-chunk.
type Stream struct {
t *Tokenizer
// count of tokens in the stream
len int
// pointer to the node of double-linked list of tokens
current *Token
// pointer of valid token if current moved to out of bounds (out of end list)
prev *Token
// pointer of valid token if current moved to out of bounds (out of begin list)
next *Token
// pointer to head of list
head *Token
// last whitespaces before end of source
wsTail []byte
// count of parsed bytes
parsed int
p *parsing
historySize int
}
// NewStream creates new parsed stream of tokens.
func NewStream(p *parsing) *Stream {
return &Stream{
t: p.t,
head: p.head,
current: p.head,
len: p.n,
wsTail: p.tail,
parsed: p.parsed + p.pos,
}
}
// NewInfStream creates new stream with active parser.
func NewInfStream(p *parsing) *Stream {
return &Stream{
t: p.t,
p: p,
len: p.n,
head: p.head,
current: p.head,
}
}
// SetHistorySize sets the number of tokens that should remain after the current token
func (s *Stream) SetHistorySize(size int) *Stream {
s.historySize = size
return s
}
// Close releases all token objects to pool
func (s *Stream) Close() {
for ptr := s.head; ptr != nil; {
p := ptr.next
s.t.freeToken(ptr)
ptr = p
}
s.next = nil
s.prev = nil
s.head = undefToken
s.current = undefToken
s.len = 0
}
func (s *Stream) String() string {
items := make([]string, 0, s.len)
ptr := s.head
for ptr != nil {
items = append(items, strconv.Itoa(ptr.id)+": "+ptr.String())
ptr = ptr.next
}
return strings.Join(items, "\n")
}
// GetParsedLength returns currently count parsed bytes.
func (s *Stream) GetParsedLength() int {
if s.p == nil {
return s.parsed
} else {
return s.p.parsed + s.p.pos
}
}
// GoNext moves stream pointer to the next token.
// If there is no token, it initiates the parsing of the next chunk of data.
// If there is no data, the pointer will point to the TokenUndef token.
func (s *Stream) GoNext() *Stream {
if s.current.next != nil {
s.current = s.current.next
if s.current.next == nil && s.p != nil { // lazy load and parse next data-chunk
n := s.p.n
s.p.parse()
s.len += s.p.n - n
}
if s.historySize != 0 && s.current.id-s.head.id > s.historySize {
t := s.head
s.head = s.head.unlink()
s.t.freeToken(t)
s.len--
}
} else if s.current == undefToken {
s.current = s.prev
s.prev = nil
} else {
s.prev = s.current
s.current = undefToken
}
return s
}
// GoPrev moves pointer of stream to the next token.
// The number of possible calls is limited if you specified SetHistorySize.
// If the beginning of the stream or the end of the history is reached, the pointer will point to the TokenUndef token.
func (s *Stream) GoPrev() *Stream {
if s.current.prev != nil {
s.current = s.current.prev
} else if s.current == undefToken {
s.current = s.next
s.prev = nil
} else {
s.next = s.current
s.current = undefToken
}
return s
}
// GoTo moves pointer of stream to specific token.
// The search is done by token ID.
func (s *Stream) GoTo(id int) *Stream {
if id > s.current.id {
for s.current != nil && id != s.current.id {
s.GoNext()
}
} else if id < s.current.id {
for s.current != nil && id != s.current.id {
s.GoPrev()
}
}
return s
}
// IsValid checks if stream is valid.
// This means that the pointer has not reached the end of the stream.
func (s *Stream) IsValid() bool {
return s.current != undefToken
}
// IsNextSequence checks if these are next tokens in exactly the same sequence as specified.
func (s *Stream) IsNextSequence(keys ...TokenKey) bool {
var (
result = true
hSize = 0
id = s.CurrentToken().ID()
)
if s.historySize > 0 && s.historySize < len(keys) {
hSize = s.historySize
s.historySize = len(keys)
}
for _, key := range keys {
if !s.GoNext().CurrentToken().Is(key) {
result = false
break
}
}
s.GoTo(id)
if hSize != 0 {
s.SetHistorySize(hSize)
}
return result
}
// IsAnyNextSequence checks that at least one token from each group is contained in a sequence of tokens
func (s *Stream) IsAnyNextSequence(keys ...[]TokenKey) bool {
var (
result = true
hSize = 0
id = s.CurrentToken().ID()
)
if s.historySize > 0 && s.historySize < len(keys) {
hSize = s.historySize
s.historySize = len(keys)
}
for _, key := range keys {
found := false
for _, k := range key {
if s.GoNext().CurrentToken().Is(k) {
found = true
break
}
}
if !found {
result = false
break
}
}
s.GoTo(id)
if hSize != 0 {
s.SetHistorySize(hSize)
}
return result
}
// HeadToken returns pointer to head-token
// Head-token may be changed if history size set.
func (s *Stream) HeadToken() *Token {
return s.head
}
// CurrentToken always returns the token.
// If the pointer is not valid (see IsValid) CurrentToken will be returns TokenUndef token.
// Do not save result (Token) into variables — current token may be changed at any time.
func (s *Stream) CurrentToken() *Token {
return s.current
}
// PrevToken returns previous token from the stream.
// If previous token doesn't exist method return TypeUndef token.
// Do not save result (Token) into variables — previous token may be changed at any time.
func (s *Stream) PrevToken() *Token {
if s.current.prev != nil {
return s.current.prev
}
return undefToken
}
// NextToken returns next token from the stream.
// If next token doesn't exist method return TypeUndef token.
// Do not save result (Token) into variables — next token may be changed at any time.
func (s *Stream) NextToken() *Token {
if s.current.next != nil {
return s.current.next
}
return undefToken
}
// GoNextIfNextIs moves stream pointer to the next token if the next token has specific token keys.
// If keys matched pointer will be updated and method returned true. Otherwise, returned false.
func (s *Stream) GoNextIfNextIs(key TokenKey, otherKeys ...TokenKey) bool {
if s.NextToken().Is(key, otherKeys...) {
s.GoNext()
return true
}
return false
}
// GetSnippet returns slice of tokens.
// Slice generated from current token position and include tokens before and after current token.
func (s *Stream) GetSnippet(before, after int) []Token {
var segment []Token
if s.current == undefToken {
if s.prev != nil && before > s.prev.id-s.head.id {
before = s.prev.id - s.head.id
} else {
before = 0
}
} else if before > s.current.id-s.head.id {
before = s.current.id - s.head.id
}
if after > s.len-before-1 {
after = s.len - before - 1
}
segment = make([]Token, before+after+1)
var ptr *Token
if s.next != nil {
ptr = s.next
} else if s.prev != nil {
ptr = s.prev
} else {
ptr = s.current
}
for p := ptr; p != nil; p, before = ptr.prev, before-1 {
segment[before] = Token{
id: ptr.id,
key: ptr.key,
value: ptr.value,
line: ptr.line,
offset: ptr.offset,
indent: ptr.indent,
string: ptr.string,
}
if before <= 0 {
break
}
}
for p, i := ptr.next, 1; p != nil; p, i = p.next, i+1 {
segment[before+i] = Token{
id: p.id,
key: p.key,
value: p.value,
line: p.line,
offset: p.offset,
indent: p.indent,
string: p.string,
}
if i >= after {
break
}
}
return segment
}
// GetSnippetAsString returns tokens before and after current token as string.
// `maxStringLength` specify max length of each token string. Zero — unlimited token string length.
// If string greater than maxLength method removes some runes in the middle of the string.
func (s *Stream) GetSnippetAsString(before, after, maxStringLength int) string {
segments := s.GetSnippet(before, after)
str := make([]string, len(segments))
for i, token := range segments {
v := token.ValueString()
if maxStringLength > 4 && len(v) > maxStringLength {
str[i] = v[:maxStringLength/2] + "..." + v[maxStringLength/2+1:]
} else {
str[i] = v
}
}
return strings.Join(str, "")
}
package tokenizer
import (
"fmt"
"strconv"
)
var undefToken = &Token{
id: -1,
}
// Token struct describe one token.
type Token struct {
id int
key TokenKey
value []byte
line int
offset int
indent []byte
string *StringSettings
prev *Token
next *Token
}
// addNext add new token as next node of dl-list.
func (t *Token) addNext(next *Token) {
next.prev = t
t.next = next
}
// unlink remove token from dl-list and fix links of prev and next nodes.
// Method returns next token or nil if no next token found.
func (t *Token) unlink() *Token {
next := t.next
t.next.prev = nil
t.next = nil
t.prev = nil
return next
}
// ID returns id of token. Id is the sequence number of tokens in the stream.
func (t *Token) ID() int {
return t.id
}
// String returns a multiline string with the token's information.
func (t Token) String() string {
return fmt.Sprintf("{\n\tId: %d\n\tKey: %d\n\tValue: %s\n\tPosition: %d\n\tIndent: %d bytes\n\tLine: %d\n}",
t.id, t.key, t.value, t.offset, len(t.indent), t.line)
}
// IsValid checks if this token is valid — the key is not TokenUndef.
func (t *Token) IsValid() bool {
return t.key != TokenUndef
}
// IsKeyword checks if this is keyword — the key is TokenKeyword.
func (t Token) IsKeyword() bool {
return t.key == TokenKeyword
}
// IsNumber checks if this token is integer or float — the key is TokenInteger or TokenFloat.
func (t Token) IsNumber() bool {
return t.key == TokenInteger || t.key == TokenFloat
}
// IsFloat checks if this token is float — the key is TokenFloat.
func (t Token) IsFloat() bool {
return t.key == TokenFloat
}
// IsInteger checks if this token is integer — the key is TokenInteger.
func (t Token) IsInteger() bool {
return t.key == TokenInteger
}
// ValueInt returns value as int64.
// If the token is float the result wild be round by math's rules.
// If the token is not TokenInteger or TokenFloat zero will be returned.
// Method doesn't use cache. Each call starts a number parser.
func (t Token) ValueInt() int64 {
if t.key == TokenInteger {
num, _ := strconv.ParseInt(b2s(t.value), 10, 64)
return num
} else if t.key == TokenFloat {
num, _ := strconv.ParseFloat(b2s(t.value), 64)
return int64(num)
}
return 0
}
// ValueFloat returns value as float64.
// If the token is not TokenInteger or TokenFloat zero will be returned.
// Method doesn't use cache. Each call starts a number parser.
func (t *Token) ValueFloat() float64 {
if t.key == TokenFloat {
num, _ := strconv.ParseFloat(b2s(t.value), 64)
return num
} else if t.key == TokenInteger {
num, _ := strconv.ParseInt(b2s(t.value), 10, 64)
return float64(num)
}
return 0.0
}
// Indent returns spaces before the token.
func (t *Token) Indent() []byte {
return t.indent
}
// Key returns the key of the token pointed to by the pointer.
// If pointer is not valid (see IsValid) TokenUndef will be returned.
func (t *Token) Key() TokenKey {
return t.key
}
// Value returns value of current token as slice of bytes from source.
// If current token is invalid value returns nil.
//
// Do not change bytes in the slice. Copy slice before change.
func (t *Token) Value() []byte {
return t.value
}
// ValueString returns value of the token as string.
// If the token is TokenUndef method returns empty string.
func (t *Token) ValueString() string {
if t.value == nil {
return ""
}
return b2s(t.value)
}
// Line returns line number in input string.
// Line numbers starts from 1.
func (t *Token) Line() int {
return t.line
}
// Offset returns the byte position in input string (from start).
func (t *Token) Offset() int {
return t.offset
}
// StringSettings returns StringSettings structure if token is framed string.
func (t *Token) StringSettings() *StringSettings {
return t.string
}
// StringKey returns key of string.
// If key not defined for string TokenString will be returned.
func (t *Token) StringKey() TokenKey {
if t.string != nil {
return t.string.Key
}
return TokenString
}
// IsString checks if current token is a quoted string.
// Token key may be TokenString or TokenStringFragment.
func (t Token) IsString() bool {
return t.key == TokenString || t.key == TokenStringFragment
}
// ValueUnescaped returns clear (unquoted) string
// - without edge-tokens (quotes)
// - with character escaping handling
//
// For example quoted string
// "one \"two\"\t three"
// transforms to
// one "two" three
// Method doesn't use cache. Each call starts a string parser.
func (t *Token) ValueUnescaped() []byte {
if t.string != nil {
from := 0
to := len(t.value)
if bytesStarts(t.string.StartToken, t.value) {
from = len(t.string.StartToken)
}
if bytesEnds(t.string.EndToken, t.value) {
to = len(t.value) - len(t.string.EndToken)
}
str := t.value[from:to]
result := make([]byte, 0, len(str))
escaping := false
start := 0
for i := 0; i < len(str); i++ {
if escaping {
if v, ok := t.string.SpecSymbols[str[i]]; ok {
result = append(result, t.value[start:i]...)
result = append(result, v)
}
start = i
escaping = false
} else if t.string.EscapeSymbol != 0 && str[i] == t.string.EscapeSymbol {
escaping = true
}
}
if start == 0 { // no one escapes
return str
}
return result
}
return t.value
}
// ValueUnescapedString like as ValueUnescaped but returns string.
func (t *Token) ValueUnescapedString() string {
if s := t.ValueUnescaped(); s != nil {
return b2s(s)
}
return ""
}
// Is checks if the token has any of these keys.
func (t *Token) Is(key TokenKey, keys ...TokenKey) bool {
if t.key == key {
return true
}
if len(keys) > 0 {
for _, k := range keys {
if t.key == k {
return true
}
}
}
return false
}
package tokenizer
import (
"io"
"sort"
"sync"
)
const newLine = '\n'
// TokenKey token type identifier
type TokenKey int
const (
// TokenUnknown means that this token not embedded token and not user defined.
TokenUnknown TokenKey = -6
// TokenStringFragment means that this is only fragment of quoted string with injections
// For example, "one {{ two }} three", where "one " and " three" — TokenStringFragment
TokenStringFragment TokenKey = -5
// TokenString means than this token is quoted string.
// For example, "one two"
TokenString TokenKey = -4
// TokenFloat means that this token is float number with point and/or exponent.
// For example, 1.2, 1e6, 1E-6
TokenFloat TokenKey = -3
// TokenInteger means that this token is integer number.
// For example, 3, 49983
TokenInteger TokenKey = -2
// TokenKeyword means that this token is word.
// For example, one, two, три
TokenKeyword TokenKey = -1
// TokenUndef means that token doesn't exist.
// Then stream out of range of token list any getter or checker will return TokenUndef token.
TokenUndef TokenKey = 0
)
const (
fStopOnUnknown uint16 = 0b1
fAllowKeywordUnderscore uint16 = 0b10
fAllowNumberUnderscore uint16 = 0b100
fAllowNumberInKeyword uint16 = 0b1000
)
// BackSlash just backslash byte
const BackSlash = '\\'
var defaultWhiteSpaces = []byte{' ', '\t', '\n', '\r'}
// DefaultStringEscapes is default escaped symbols. Those symbols are often used everywhere.
var DefaultStringEscapes = map[byte]byte{
'n': '\n',
'r': '\r',
't': '\t',
'\\': '\\',
}
// tokenItem describes one token.
type tokenRef struct {
// Token type. Not unique.
Key TokenKey
// Token value as is. Should be unique.
Token []byte
}
// QuoteInjectSettings describes open injection token and close injection token.
type QuoteInjectSettings struct {
// Token type witch opens quoted string.
StartKey TokenKey
// Token type witch closes quoted string.
EndKey TokenKey
}
// StringSettings describes framed(quoted) string tokens like quoted strings.
type StringSettings struct {
Key TokenKey
StartToken []byte
EndToken []byte
EscapeSymbol byte
SpecSymbols map[byte]byte
Injects []QuoteInjectSettings
}
// AddInjection configure injection in to string.
// Injection - parsable fragment of framed(quoted) string.
// Often used for parsing of placeholders or template's expressions in the framed string.
func (q *StringSettings) AddInjection(startTokenKey, endTokenKey TokenKey) *StringSettings {
q.Injects = append(q.Injects, QuoteInjectSettings{StartKey: startTokenKey, EndKey: endTokenKey})
return q
}
// SetEscapeSymbol set escape symbol for framed(quoted) string.
// Escape symbol allows ignoring close token of framed string.
// Also escape symbol allows using special symbols in the frame strings, like \n, \t.
func (q *StringSettings) SetEscapeSymbol(symbol byte) *StringSettings {
q.EscapeSymbol = symbol
return q
}
// SetSpecialSymbols set mapping of all escapable symbols for escape symbol, like \n, \t, \r.
func (q *StringSettings) SetSpecialSymbols(special map[byte]byte) *StringSettings {
q.SpecSymbols = special
return q
}
// Tokenizer stores all tokens configuration and behaviors.
type Tokenizer struct {
// bit flags
flags uint16
// all defined custom tokens {key: [token1, token2, ...], ...}
tokens map[TokenKey][]*tokenRef
index map[byte][]*tokenRef
quotes []*StringSettings
wSpaces []byte
pool sync.Pool
}
// New creates new tokenizer.
func New() *Tokenizer {
t := Tokenizer{
flags: 0,
tokens: map[TokenKey][]*tokenRef{},
index: map[byte][]*tokenRef{},
quotes: []*StringSettings{},
wSpaces: defaultWhiteSpaces,
}
t.pool.New = func() interface{} {
return new(Token)
}
return &t
}
// SetWhiteSpaces sets custom whitespace symbols between tokens.
// By default: {' ', '\t', '\n', '\r'}
func (t *Tokenizer) SetWhiteSpaces(ws []byte) *Tokenizer {
t.wSpaces = ws
return t
}
// StopOnUndefinedToken stops parsing if unknown token detected.
func (t *Tokenizer) StopOnUndefinedToken() *Tokenizer {
t.flags |= fStopOnUnknown
return t
}
// AllowKeywordUnderscore allows underscore symbol in keywords, like `one_two` or `_three`
func (t *Tokenizer) AllowKeywordUnderscore() *Tokenizer {
t.flags |= fAllowKeywordUnderscore
return t
}
// AllowNumbersInKeyword allows numbers in keywords, like `one1` or `r2d2`
// The method allows numbers in keywords, but the keyword itself must not start with a number.
// There should be no spaces between letters and numbers.
func (t *Tokenizer) AllowNumbersInKeyword() *Tokenizer {
t.flags |= fAllowNumberInKeyword
return t
}
// DefineTokens add custom token.
// There `key` unique is identifier of `tokens`, `tokens` — slice of string of tokens.
// If key already exists tokens will be rewritten.
func (t *Tokenizer) DefineTokens(key TokenKey, tokens []string) *Tokenizer {
var tks []*tokenRef
if key < 1 {
return t
}
for _, token := range tokens {
ref := tokenRef{
Key: key,
Token: s2b(token),
}
head := ref.Token[0]
tks = append(tks, &ref)
if t.index[head] == nil {
t.index[head] = []*tokenRef{}
}
t.index[head] = append(t.index[head], &ref)
sort.Slice(t.index[head], func(i, j int) bool {
return len(t.index[head][i].Token) > len(t.index[head][j].Token)
})
}
t.tokens[key] = tks
return t
}
// DefineStringToken defines a token string.
// For example, a piece of data surrounded by quotes: "string in quotes" or 'string on sigle quotes'.
// Arguments startToken and endToken defines open and close "quotes".
// - t.DefineStringToken("`", "`") - parse string "one `two three`" will be parsed as
// [{key: TokenKeyword, value: "one"}, {key: TokenString, value: "`two three`"}]
// - t.DefineStringToken("//", "\n") - parse string "parse // like comment\n" will be parsed as
// [{key: TokenKeyword, value: "parse"}, {key: TokenString, value: "// like comment"}]
func (t *Tokenizer) DefineStringToken(key TokenKey, startToken, endToken string) *StringSettings {
q := &StringSettings{
Key: key,
StartToken: s2b(startToken),
EndToken: s2b(endToken),
}
if q.StartToken == nil {
return q
}
t.quotes = append(t.quotes, q)
return q
}
func (t *Tokenizer) allocToken() *Token {
return t.pool.Get().(*Token)
}
func (t *Tokenizer) freeToken(token *Token) {
token.next = nil
token.prev = nil
token.value = nil
token.indent = nil
token.offset = 0
token.line = 0
token.id = 0
token.key = 0
token.string = nil
t.pool.Put(token)
}
// ParseString parse the string into tokens
func (t *Tokenizer) ParseString(str string) *Stream {
return t.ParseBytes(s2b(str))
}
// ParseBytes parse the bytes slice into tokens
func (t *Tokenizer) ParseBytes(str []byte) *Stream {
p := newParser(t, str)
p.parse()
return NewStream(p)
}
// ParseStream parse the string into tokens.
func (t *Tokenizer) ParseStream(r io.Reader, bufferSize uint) *Stream {
p := newInfParser(t, r, bufferSize)
p.preload()
p.parse()
return NewInfStream(p)
}
# Created by https://www.toptal.com/developers/gitignore/api/intellij,go
# Edit at https://www.toptal.com/developers/gitignore?templates=intellij,go
### Go ###
# If you prefer the allow list template instead of the deny list, see community template:
# https://github.com/github/gitignore/blob/main/community/Golang/Go.AllowList.gitignore
#
# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
# Test binary, built with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out
testdata/
# Dependency directories (remove the comment below to include it)
# vendor/
# Go workspace file
go.work
# Go Fuzz build
testdata/
### Go Patch ###
/vendor/
/Godeps/
### Intellij ###
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
# User-specific stuff
.idea/**/workspace.xml
.idea/**/tasks.xml
.idea/**/usage.statistics.xml
.idea/**/dictionaries
.idea/**/shelf
# AWS User-specific
.idea/**/aws.xml
# Generated files
.idea/**/contentModel.xml
# Sensitive or high-churn files
.idea/**/dataSources/
.idea/**/dataSources.ids
.idea/**/dataSources.local.xml
.idea/**/sqlDataSources.xml
.idea/**/dynamic.xml
.idea/**/uiDesigner.xml
.idea/**/dbnavigator.xml
# Gradle
.idea/**/gradle.xml
.idea/**/libraries
# Gradle and Maven with auto-import
# When using Gradle or Maven with auto-import, you should exclude module files,
# since they will be recreated, and may cause churn. Uncomment if using
# auto-import.
# .idea/artifacts
# .idea/compiler.xml
# .idea/jarRepositories.xml
# .idea/modules.xml
# .idea/*.iml
# .idea/modules
# *.iml
# *.ipr
# CMake
cmake-build-*/
# Mongo Explorer plugin
.idea/**/mongoSettings.xml
# File-based project format
*.iws
# IntelliJ
out/
# mpeltonen/sbt-idea plugin
.idea_modules/
# JIRA plugin
atlassian-ide-plugin.xml
# Cursive Clojure plugin
.idea/replstate.xml
# SonarLint plugin
.idea/sonarlint/
# Crashlytics plugin (for Android Studio and IntelliJ)
com_crashlytics_export_strings.xml
crashlytics.properties
crashlytics-build.properties
fabric.properties
# Editor-based Rest Client
.idea/httpRequests
# Android studio 3.1+ serialized cache file
.idea/caches/build_file_checksums.ser
### Intellij Patch ###
# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721
# *.iml
# modules.xml
# .idea/misc.xml
# *.ipr
# Sonarlint plugin
# https://plugins.jetbrains.com/plugin/7973-sonarlint
.idea/**/sonarlint/
# SonarQube Plugin
# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin
.idea/**/sonarIssues.xml
# Markdown Navigator plugin
# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced
.idea/**/markdown-navigator.xml
.idea/**/markdown-navigator-enh.xml
.idea/**/markdown-navigator/
# Cache file creation bug
# See https://youtrack.jetbrains.com/issue/JBR-2257
.idea/$CACHE_FILE$
# CodeStream plugin
# https://plugins.jetbrains.com/plugin/12206-codestream
.idea/codestream.xml
# Azure Toolkit for IntelliJ plugin
# https://plugins.jetbrains.com/plugin/8053-azure-toolkit-for-intellij
.idea/**/azureSettings.xml
# End of https://www.toptal.com/developers/gitignore/api/intellij,go
\ No newline at end of file
<a name="v1.16.0"></a>
## [v1.16.0] - 2023-05-20
<a name="v1.15.0"></a>
## [v1.15.0] - 2023-03-09
<a name="v1.14.0"></a>
## [v1.14.0] - 2023-03-09
<a name="v1.13.2"></a>
## [v1.13.2] - 2022-12-23
<a name="v1.13.1"></a>
## [v1.13.1] - 2022-12-23
<a name="v1.13.0"></a>
## [v1.13.0] - 2022-10-23
### Add Features
- feat change proxy to map and expose map
### Changes
- chore add licenses
<a name="v1.12.0"></a>
## [v1.12.0] - 2022-10-19
### Bug Fixes
- fix remove tag for command [#4](https://gitlab.schukai.com/oss/libraries/go/application/xflags/issues/4)
<a name="v1.11.0"></a>
## [v1.11.0] - 2022-10-19
### Add Features
- feat improve the help output
<a name="v1.10.2"></a>
## [v1.10.2] - 2022-10-16
### Bug Fixes
- fix copy only explicite values [#3](https://gitlab.schukai.com/oss/libraries/go/application/xflags/issues/3)
<a name="v1.10.1"></a>
## [v1.10.1] - 2022-10-16
### Bug Fixes
- fix assign the correct value to the proxy
### Changes
- chore add license texts
<a name="v1.10.0"></a>
## [v1.10.0] - 2022-10-15
### Code Refactoring
- refactor functions moved to a separate repos
<a name="v1.9.0"></a>
## [v1.9.0] - 2022-10-15
### Add Features
- feat implements proxy interface [#2](https://gitlab.schukai.com/oss/libraries/go/application/xflags/issues/2)
<a name="v1.8.3"></a>
## [v1.8.3] - 2022-10-15
### Bug Fixes
- fix help request should not removed from errors
<a name="v1.8.2"></a>
## [v1.8.2] - 2022-10-15
### Bug Fixes
- fix pathfinder panic
<a name="v1.8.1"></a>
## [v1.8.1] - 2022-10-15
<a name="v1.8.0"></a>
## [v1.8.0] - 2022-10-15
### Bug Fixes
- fix invalid type exception
<a name="v1.7.0"></a>
## [v1.7.0] - 2022-10-14
### Add Features
- feat introduction of copy-interface
- feat new method to get the output of the flags
<a name="v1.6.0"></a>
## [v1.6.0] - 2022-10-13
### Code Refactoring
- refactor the execute function, the execute function should not print messages
<a name="v1.5.0"></a>
## [v1.5.0] - 2022-10-13
### Add Features
- feat new opportunities for interaction with flags [#1](https://gitlab.schukai.com/oss/libraries/go/application/xflags/issues/1)
### Changes
- chore add license
<a name="v1.4.0"></a>
## [v1.4.0] - 2022-10-09
### Add Features
- feat new Execute() Method for the automatic execution of command
- feat new function GetDefaults()
<a name="v1.3.1"></a>
## [v1.3.1] - 2022-10-08
### Bug Fixes
- fix remove one test renmant
<a name="v1.3.0"></a>
## [v1.3.0] - 2022-10-08
### Code Refactoring
- refactor change func name FlagOutput() to Output()
<a name="v1.2.3"></a>
## [v1.2.3] - 2022-10-07
### Changes
- chore change license things
- chore change license things
<a name="v1.2.2"></a>
## [v1.2.2] - 2022-10-06
### Changes
- chore add licenses
<a name="v1.2.1"></a>
## [v1.2.1] - 2022-10-06
### Changes
- chore add licenses header
<a name="v1.2.0"></a>
## [v1.2.0] - 2022-10-05
### Add Features
- feat new function ParseOsArgs
### Bug Fixes
- fix Settings should be exported
- fix parse only the arguments and not the programm
<a name="v1.1.1"></a>
## [v1.1.1] - 2022-10-05
### Bug Fixes
- fix repository_url was wrong
<a name="v1.1.0"></a>
## [v1.1.0] - 2022-10-05
<a name="v1.0.0"></a>
## v1.0.0 - 2022-10-04
[v1.16.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.15.0...v1.16.0
[v1.15.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.14.0...v1.15.0
[v1.14.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.13.2...v1.14.0
[v1.13.2]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.13.1...v1.13.2
[v1.13.1]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.13.0...v1.13.1
[v1.13.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.12.0...v1.13.0
[v1.12.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.11.0...v1.12.0
[v1.11.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.10.2...v1.11.0
[v1.10.2]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.10.1...v1.10.2
[v1.10.1]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.10.0...v1.10.1
[v1.10.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.9.0...v1.10.0
[v1.9.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.8.3...v1.9.0
[v1.8.3]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.8.2...v1.8.3
[v1.8.2]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.8.1...v1.8.2
[v1.8.1]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.8.0...v1.8.1
[v1.8.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.7.0...v1.8.0
[v1.7.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.6.0...v1.7.0
[v1.6.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.5.0...v1.6.0
[v1.5.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.4.0...v1.5.0
[v1.4.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.3.1...v1.4.0
[v1.3.1]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.3.0...v1.3.1
[v1.3.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.2.3...v1.3.0
[v1.2.3]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.2.2...v1.2.3
[v1.2.2]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.2.1...v1.2.2
[v1.2.1]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.2.0...v1.2.1
[v1.2.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.1.1...v1.2.0
[v1.1.1]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.1.0...v1.1.1
[v1.1.0]: https://gitlab.schukai.com/oss/libraries/go/application/xflags/compare/v1.0.0...v1.1.0
## Copyright 2022 schukai GmbH. All rights reserved.
## Use of this source code is governed by a AGPL-3.0
## license that can be found in the LICENSE file.
PROJECT_ROOT:=$(dir $(realpath $(lastword $(MAKEFILE_LIST))))
THIS_MAKEFILE:=$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST))
THIS_MAKEFILE_PATH:=$(PROJECT_ROOT)$(THIS_MAKEFILE)
# @see .PHONY https://www.gnu.org/software/make/manual/html_node/Phony-Targets.html#Phony-Targets
.DEFAULT_GOAL := help
.PHONY: print
## Print Path
print:
@echo "THIS_MAKEFILE: $(THIS_MAKEFILE)"
@echo "THIS_MAKEFILE_PATH: $(THIS_MAKEFILE_PATH)"
@echo "PROJECT_ROOT: $(PROJECT_ROOT)"
# Add a comment to the public targets so that it appears
# in this help Use two # characters for a help comment
.PHONY: help
help:
@printf "${COMMENT}Usage:${RESET}\n"
@printf " make [target] [arg=\"val\"...]\n\n"
@printf "${COMMENT}Available targets:${RESET}\n"
@awk '/^[a-zA-Z\-\\_0-9\.@]+:/ { \
helpMessage = match(lastLine, /^## (.*)/); \
if (helpMessage) { \
helpCommand = substr($$1, 0, index($$1, ":")); \
helpMessage = substr(lastLine, RSTART + 3, RLENGTH); \
printf " ${INFO}%-22s${RESET} %s\n", helpCommand, helpMessage; \
} \
} \
{ lastLine = $$0 }' $(MAKEFILE_LIST)
@printf "\n${COMMENT}Available arguments:${RESET}\n\n"
@awk '/^(([a-zA-Z\-\\_0-9\.@]+)\s[?:]?=)/ { \
helpMessage = match(lastLine, /^## (.*)/); \
if (helpMessage) { \
helpMessage = substr(lastLine, RSTART + 3, RLENGTH); \
printf " ${INFO}%-22s${RESET} %s (Default: %s)\n", $$1, helpMessage, $$3; \
} \
} \
{ lastLine = $$0 }' $(MAKEFILE_LIST)
## run tests
test:
echo "Running tests"
go test -cover -v ./...
## run tests with fuzzing
test-fuzz:
echo "Running fuzz tests"
go test -v -fuzztime=30s -fuzz=Fuzz ./...
#### VERSION
BIN_DIR ?= $(shell echo $$HOME)/.local/bin/
VERSION_NAME := version
EXECUTABLES = $(EXECUTABLES:-) $(VERSION_NAME)
VERSION_BIN_PATH := $(BIN_DIR)$(VERSION_NAME)
VERSION_BIN := $(shell command -v $(VERSION_NAME) 2> /dev/null)
ifndef VERSION_BIN
$(shell curl -o $(VERSION_BIN_PATH) http://download.schukai.com/tools/version/version-$(shell uname -s | tr [:upper:] [:lower:])-$(shell echo `uname -m | sed s/aarch64/arm64/ | sed s/x86_64/amd64/`))
$(shell chmod +x $(VERSION_BIN_PATH))
endif
GIT_CHGLOG_BIN := $(shell command -v git-chglog 2> /dev/null)
ifeq ($(GIT_CHGLOG_BIN),)
$(shell go install github.com/git-chglog/git-chglog/cmd/git-chglog@latest)
endif
RELEASE_FILE ?= $(PROJECT_ROOT)release.json
CHANGELOG_FILE ?= $(PROJECT_ROOT)CHANGELOG.md
ifeq ("$(wildcard $(RELEASE_FILE))","")
$(shell echo '{"version":"0.1.0"}' > $(RELEASE_FILE))
endif
PROJECT_VERSION ?= $(shell cat $(RELEASE_FILE) | jq -r .version)
PROJECT_BUILD_DATE ?= $(shell $(VERSION_BIN) date)
.PHONY: next-patch-version
next-patch-version: check-clean-repo
echo "Creating next version"
$(VERSION_BIN) patch --path $(RELEASE_FILE) --selector "version"
git add $(RELEASE_FILE) && git commit -m "Bump version to $$(cat $(RELEASE_FILE) | jq -r .version)"
.PHONY: next-minor-version
next-minor-version: check-clean-repo
echo "Creating next minor version"
$(VERSION_BIN) minor --path $(RELEASE_FILE) --selector "version"
git add $(RELEASE_FILE) && git commit -m "Bump version to $$( cat $(RELEASE_FILE) | jq -r .version)"
.PHONY: next-major-version
next-major-version: check-clean-repo
echo "Creating next minor version"
$(VERSION_BIN) major --path $(RELEASE_FILE) --selector "version"
git add $(RELEASE_FILE) && git commit -m "Bump version to $$(cat $(RELEASE_FILE) | jq -r .version)"
.PHONY: check-clean-repo
check-clean-repo:
git diff-index --quiet HEAD || (echo "There are uncommitted changes after running make. Please commit or stash them before running make."; exit 1)
## tag repository with next patch version
tag-patch-version: next-patch-version
echo "Tagging patch version"
$(eval PROJECT_VERSION := $(shell cat $(RELEASE_FILE) | jq -r .version))
git-chglog --next-tag v$(PROJECT_VERSION) -o $(CHANGELOG_FILE)
git add $(CHANGELOG_FILE) && git commit -m "Update changelog"
git tag -a v$(PROJECT_VERSION) -m "Version $(PROJECT_VERSION)"
## tag repository with next minor version
tag-minor-version: next-minor-version
echo "Tagging minor version"
$(eval PROJECT_VERSION := $(shell cat $(RELEASE_FILE) | jq -r .version))
git-chglog --next-tag v$(PROJECT_VERSION) -o $(CHANGELOG_FILE)
git add $(CHANGELOG_FILE) && git commit -m "Update changelog"
git tag -a v$(PROJECT_VERSION) -m "Version $(PROJECT_VERSION)"
## tag repository with next major version
tag-major-version: next-major-version
echo "Tagging major version"
$(eval PROJECT_VERSION := $(shell cat $(RELEASE_FILE) | jq -r .version))
git-chglog --next-tag v$(PROJECT_VERSION) -o $(CHANGELOG_FILE)
git add $(CHANGELOG_FILE) && git commit -m "Update changelog"
git tag -a v$(PROJECT_VERSION) -m "Version $(PROJECT_VERSION)"
GO_MOD_FILE := $(SOURCE_PATH)go.mod
ifeq ($(shell test -e $(GO_MOD_FILE) && echo -n yes),yes)
GO_CURRENT_MODULE := $(shell cat $(GO_MOD_FILE) | head -n1 | cut -d" " -f2)
# go install github.com/google/go-licenses@latest
EXECUTABLES = $(EXECUTABLES:-) go-licenses;
endif
.PHONY: fetch-licenses
## Fetch licenses for all modules
fetch-licenses:
go-licenses save $(GO_CURRENT_MODULE) --ignore gitlab.schukai.com --force --save_path $(PROJECT_ROOT)licenses/
# https://spdx.github.io/spdx-spec/v2.3/SPDX-license-list/
ADDLICENSE_BIN ?= addlicense
ifeq ($(shell command -v $(ADDLICENSE_BIN) 2> /dev/null),)
$(shell go install github.com/google/addlicense@latest)
EXECUTABLES = $(EXECUTABLES:-) $(ADDLICENSE_BIN);
endif
.PHONY: add-licenses
## Add license headers to all go files
add-licenses:
addlicense -c "schukai GmbH" -s -l "AGPL-3.0" ./*.go
## X-Flags
## What does this library?
This library provides a simple way to use flags in your application. It extends the standard library
to be able to define and use a structure with flags.
It supports:
* [X] Define flags in a structure
* [X] Define callbacks for flags
* [X] Define default values for flags
* [X] Define a map for values
## Installation
```shell
go get gitlab.schukai.com/oss/libraries/go/application/xflags
```
**Note:** This library uses [Go Modules](https://github.com/golang/go/wiki/Modules) to manage dependencies.
## Usage
### Initialize
A new flag set is created using the `xflags.New()` function.
The structure passed is used as the type for the flags.
```go
package main
import (
"fmt"
"os"
"gitlab.schukai.com/oss/libraries/go/application/xflags"
)
```
### Definition
The flags are defined in the structure. The structure can be nested.
The name of the field is used as the name of the flag. The type of the
field is used as the type of the flag.
```go
type Definition struct {
Verbose bool `short:"v" long:"verbose" description:"Show verbose debug information"`
Serve struct {
Host string `short:"h" long:"host" description:"Host to bind to" default:"localhost"`
Port int `short:"p" long:"port" description:"Port to bind to" default:"8080"`
} `command:"serve" description:"Run the HTTP server" call:"DoServe"`
}
```
The following tags are supported:
| Tag | Context | Description |
|---------------|----------|--------------------------------------------|
| `short` | Value | Short name of the flag. |
| `long` | Value | Long name of the flag. |
| `description` | Value | Description of the flag. |
| `required` | Value | Flag is required. |
| `map` | Value | Copy the value to the mapped structure. |
| `command` | Command | Flag is a command. |
| `call` | Command | Function to call when the command is used. |
| `ignore` | -/- | Property is ignored. |
### Callbacks
The functions are called up with a receiver. The receiver is the
configuration. The function must have the following signature:
`func (d *Definition) <name> (s *setting[Definition])`
Let's assume we have the above definition. The Property `Serve` contains
the command `serve`. Furthermore, the command has the tag `call` with
the value `DoServe`. The function `DoServe` is called when the command
`serve` is used.
Important: The function must be exported, that means it
must start with a capital letter.
The function is called with the receiver `*Definition`
An example for the function `DoServe`:
```go
func (d *Definition) DoServe(_ *setting[Definition]) {
fmt.Printf("Serving on %s:%d", d.Serve.Host, d.Serve.Port)
}
```
In this example, the function is called with the receiver `*Definition`.
The function is called with the setting `*setting[Definition]`. The
setting is used to get the values of the flags. But in this example, we
don't need the setting. So we use the underscore `_` to ignore the
setting.
### New Setting
The function `New` creates a new setting for the given
definition. The function returns a pointer to the setting.
The first argument is a name for the setting. The second argument is the
definition.
A good choice for the name is the argument `os.Args[0]`.
```go
// define instance
var instance *xflags.Settings[Definition]
func Execute() {
instance := xflags.New(os.Args[0], Definition{})
if instance.HasErrors() {
// ...
```
### Parse
The flags are parsed using the `Parse()` function. The function returns
the command and the setting. The command is the name of the command
which was used. The setting is the setting of the flags.
```go
setting.Parse(os.Args[1:])
```
For testing, you can use the following arguments:
```go
setting.Parse([]string{"--verbose", "serve", "--host", "localhost", "--port", "8080"})
```
### Get Values
The values of the flags are available in the setting. The values are
available in the structure. The structure is the same as the definition.
```go
fmt.Printf("Host: %s", setting.GetValues().Serve.Host)
fmt.Printf("Port: %d", setting.GetValues().Serve.Port)
```
### Execute
The function `Execute()` executes the command. See the section
[Callbacks](#callbacks) for more information.
```go
setting.Execute()
```
### Mapped Values
The mapped structure is used to copy the
values of the flags to another structure
and to a map.
The mapped structure must implement the `Copyable` interface.
```go
type MyObj struct {
Verbose bool
Serve struct {
Host string
Port int
}
}
func (m *MyObj) Copy(_ map[string]any) {
}
func main() {
setting := New(os.Args[0], Definition{})
setting.SetMappedObject(&MyObj{})
setting.Parse(os.Args[1:])
setting.Execute()
}
```
The path in the structure is defined by the tag `map`.
Die Map der Werte kann über die Methode `GetMap()` abgerufen werden.
### Arguments
the free arguments can be fetched with the method `Args()`.
### Check Status
The execution result can be queried with the functions:
- `HelpRequested() bool`
- `WasExecuted() bool`
- `Error() error`
- `MissingCommand() bool`
## Contributing
Merge requests are welcome. For major changes, please open an issue first to discuss what
you would like to change. **Please make sure to update tests as appropriate.**
Versioning is done with [SemVer](https://semver.org/).
Changelog is generated with [git-chglog](https://github.com/git-chglog/git-chglog#git-chglog)
Commit messages should follow the [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) specification.
Messages are started with a type, which is one of the following:
- **feat**: A new feature
- **fix**: A bug fix
- **doc**: Documentation only changes
- **refactor**: A code change that neither fixes a bug nor adds a feature
- **perf**: A code change that improves performance
- **test**: Adding missing or correcting existing tests
- **chore**: Other changes that don't modify src or test files
The footer would be used for a reference to an issue or a breaking change.
A commit that has a footer `BREAKING CHANGE:`, or appends a ! after the type/scope,
introduces a breaking API change (correlating with MAJOR in semantic versioning).
A BREAKING CHANGE can be part of commits of any type.
the following is an example of a commit message:
```text
feat: add 'extras' field
```
## License
[AGPL-3.0](https://choosealicense.com/licenses/agpl-3.0/)
// Copyright 2022 schukai GmbH
// SPDX-License-Identifier: AGPL-3.0
package xflags
import (
"bytes"
"flag"
"fmt"
"io"
"os"
"reflect"
)
type dummyCopyArg struct{}
func (n dummyCopyArg) Copy(_ map[string]any) {}
// Execute executes the command line arguments and calls the functions.
func Execute[C any](cmd C, cpy ...Copyable) *Settings[C] {
if cpy == nil {
return execute(cmd, dummyCopyArg{}, os.Args[0], os.Args[1:])
}
if len(cpy) > 1 {
panic("too many arguments")
}
return execute(cmd, cpy[0], os.Args[0], os.Args[1:])
}
// PrintFlagOutput prints the flag output to the standard output.
func (s *Settings[C]) PrintFlagOutput() {
fmt.Println(s.command.flagSet.Output())
}
// GetFlagOutput returns the flag output.
func (s *Settings[C]) GetFlagOutput() {
fmt.Println(s.command.flagSet.Output())
}
// execute is the internal implementation of Execute.
func execute[C any, D Copyable](cmd C, proxy D, name string, args []string) *Settings[C] {
instance := New(name, cmd)
if instance.HasErrors() {
return instance
}
if (reflect.ValueOf(&proxy).Elem().Type() != reflect.TypeOf(dummyCopyArg{})) {
instance.SetMappedObject(proxy)
if instance.HasErrors() {
return instance
}
}
instance.Parse(args)
if instance.HelpRequested() {
return instance
}
if instance.HasErrors() {
return instance
}
instance.Execute()
if instance.HasErrors() {
return instance
}
return instance
}
// New creates a new instance of the settings.
// name should be the name of the command and comes from the first argument of the command line.
// os.Args[0] is a good choice.
func New[C any](name string, definitions C) *Settings[C] {
s := &Settings[C]{
config: config{
errorHandling: flag.ContinueOnError,
},
}
if reflect.TypeOf(definitions).Kind() != reflect.Struct {
s.errors = append(s.errors, newUnsupportedReflectKindError(reflect.TypeOf(definitions)))
return s
}
s.mapping = make(map[string]any)
buf := bytes.NewBufferString("")
s.flagOutput = io.Writer(buf)
s.definitions = definitions
s.initCommands(name)
return s
}
// Output returns the writer where the flag package writes its output.
func (s *Settings[C]) Output() string {
return s.flagOutput.(*bytes.Buffer).String()
}
// Args Returns not parsed arguments.
func (s *Settings[C]) Args() []string {
return s.args
}
// GetDefaults returns the default values of the settings.
func (s *Settings[C]) GetDefaults() string {
mem := s.flagOutput
s.flagOutput.(*bytes.Buffer).Reset()
s.command.flagSet.PrintDefaults()
r := s.flagOutput.(*bytes.Buffer).String()
s.flagOutput = mem
return r
}
func (s *Settings[C]) GetMap() map[string]any {
return s.mapping
}
// This package provides a simple way to create a CLI application
// with subcommands. It is based on the flags package from
// Copyright 2022 schukai GmbH
// SPDX-License-Identifier: AGPL-3.0
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published
// by the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <https://www.gnu.org/licenses/>.
package xflags
// Copyright 2023 schukai GmbH
// SPDX-License-Identifier: AGPL-3.0
package xflags
func (s *Settings[C]) SetHint(hint string) *Settings[C] {
s.hint = hint
return s
}
func (s *Settings[C]) HasHint() bool {
return s.hint != ""
}
func (s *Settings[C]) GetHint() string {
return s.hint
}
{"version":"1.16.0"}
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment