Steps to reproduce

1. Geo Lib installing test is executed on the TravisCI: https://travis-ci.org/roma-glushko/tango/builds/645156066

Expected result

All tests are executed without errors.

Actual results

There is an error during execution

This post might be helpful:

• https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
• https://github.com/maxmind/geoipupdate

There can be an issue during Geo report generation.

Preconditions

• The latest version of Tango
• Any OS (reproduced on macOS)

Steps to reproduce

• Run the latest version of Tango like this:

tango geo -l 2019-10-13-transfer.log -r 10-13-geo-report.csv

Expected result

Geo report is generated in any case and saved correctly to the file system.

Actual results

There is an error in the console:

💃 Tango is on the scene!
💃 started to generate a geo report...
💃 reading access logs...
 433.17 MiB / 433.97 MiB [============================================================]  99.81% 1m7s
💃 saving the geo report...
2019/10/14 12:03:05 ipAddress passed to Lookup cannot be nil

No reports are saved.

We need to add an ability to release Tango's builds on Snapcraft to let Ubuntu users use the tool.

References:

• https://goreleaser.com/snapcraft/
• https://snapcraft.io/first-snap/golang/macos/package

Preconditions

Log File - transfer-ssl-2019-11-21.log

Steps to reproduce

Run Report report:

tango request -l transfer-ssl-2019-11-21.log -r 11-21.request.csv

Expected result

Request Report is generated successfully.

Actual results

Error is thrown, no report is generated:

💃 Tango is on the scene!
💃 started to generate a request report...
💃 reading access logs...
 71.73 MiB / 71.91 MiB [===============================================================]  99.74% 17s
💃 saving the request report...
panic: parse %.: invalid URL escape "%."

goroutine 1 [running]:
tango/internal/usecase/report.(*RequestReportUsecase).GenerateReport(0xc0001437d0, 0x7ffeefbff600, 0x11, 0xc00b42c000, 0x233a9, 0x2771c)
	/Users/glushko/Documents/Projects/etc/tango/internal/usecase/report/request-report-usecase.go:65 +0x655
tango/internal/cli/command.RequestReportCommand(0xc0000d0b00, 0x0, 0xc0000e2b60)
	/Users/glushko/Documents/Projects/etc/tango/internal/cli/command/request-report-command.go:26 +0x3c0
github.com/urfave/cli.HandleAction(0x14469e0, 0x151dc80, 0xc0000d0b00, 0xc0000d0b00, 0x0)
	/Users/glushko/go/pkg/mod/github.com/urfave/[email protected]/app.go:514 +0xbe
github.com/urfave/cli.Command.Run(0x14e7643, 0x7, 0x0, 0x0, 0xc0000ada60, 0x1, 0x1, 0x14f7118, 0x28, 0x0, ...)
	/Users/glushko/go/pkg/mod/github.com/urfave/[email protected]/command.go:171 +0x4cd
github.com/urfave/cli.(*App).Run(0xc00011a1c0, 0xc0000b8120, 0x6, 0x6, 0x0, 0x0)
	/Users/glushko/go/pkg/mod/github.com/urfave/[email protected]/app.go:265 +0x72f
tango/internal/cli.(*TangoCli).Run(0xc000143f48, 0xc0000b8120, 0x6, 0x6)
	/Users/glushko/Documents/Projects/etc/tango/internal/cli/tango-cli.go:208 +0x50
main.main()
	/Users/glushko/Documents/Projects/etc/tango/main.go:11 +0x5d

Preconditions

1. Nginx Log in the combined format like this:

servername.com 1.1.1.1 - - [19/Nov/2019:02:55:46 +0100] "GET /server-status HTTP/1.1" 200 216081 "-" "libwww-perl/6.15" 
10.10.10.10 2.2.2.2 - - [19/Nov/2019:02:56:46 +0100] "GET /administrator HTTP/1.1" 301 245 "-" "Mozilla/5.0 (Linux; rv:55.0) Gecko/20100101 Firefox/55.0" 
10.10.10.10 3.3.3.3 - - [19/Nov/2019:02:56:46 +0100] "GET /administrator/ HTTP/1.1" 200 3314 "-" "Mozilla/5.0 (Linux; rv:55.0) Gecko/20100101 Firefox/55.0" 
servername.com 4.4.4.4 - - [19/Nov/2019:03:00:46 +0100] "GET /server-status HTTP/1.1" 200 216194 "-" "libwww-perl/6.15" 

Steps to reproduce

1. Run any report with any params on the log

Expected result

Log file is parsed and processed

Actual results

Nothing is parsed, all information is empty. However, regex pattern is still actual

Preconditions

1. Any version of Tango
2. Any OS

Steps to reproduce

1. Run journey report

Expected result

1. Journey report is generated without errors

Actual results

There is an error on saving the report:

Error on reading journey report template: open template/journey-report.tmpl: no such file or directory

Description Tango requires the creation of .tango.yaml file to be able to generate reports.

Expected Result This should be optional and only for those who want to optimize and don't remember useful repeatable configs.

Actual Result Tango requires .tango.yaml to be able to execute commands even if the file is empty

We need to make usage of Tango geo reports zero configured and don't require people to install and use MaxMind Geo library.

Reference:

• https://dev.maxmind.com/geoip/geoip2/geolite2/ - list of zipped libraries

The following parts of the Tango CLI should be documented:

• usage of filters
• report types
• examples of report outputs
• usage of .tango.yaml config file

Travis Deployment seems to be failed on Win OS only. Deployment Log:

travis_fold:start:worker_info
Worker information
hostname: ec569576-5a39-4651-b763-075e1e5f1061@1.worker-com-6dfdbf9699-mgckx.gce-production-2
version: v6.2.1 https://github.com/travis-ci/worker/tree/4e3246c044eb4915c2378ffacd0b3d3ed0136bba
instance: travis-job-7916f5f6-ffc3-4ad2-a694-e898577ea52f travis-ci-onion-1803-containers-1542208204-ad01dca (via amqp)
startup: 7.394070823s
travis_fold:end:worker_info
Secret environment variables are not obfuscated on Windows, please refer to our documentation: https://docs.travis-ci.com/user/best-practices-security
travis_time:start:0bd629e8
travis_time:end:0bd629e8:start=1568011450964546800,finish=1568011451005869300,duration=41322500,event=check_unsupported
travis_time:start:0049e318
travis_time:end:0049e318:start=1568011451051118100,finish=1568011451090383000,duration=39264900,event=apt_get_update
travis_time:start:068886f0
Disabling Windows Defender
$ powershell -Command Set-MpPreference -DisableArchiveScanning \$true
$ powershell -Command Set-MpPreference -DisableRealtimeMonitoring \$true
$ powershell -Command Set-MpPreference -DisableBehaviorMonitoring \$true
travis_time:end:068886f0:start=1568011451136593500,finish=1568011454032883700,duration=2896290200,event=disable_windows_defender
Updating gimme
travis_time:start:001e260c

travis_fold:start:git.checkout
travis_time:start:0d1a397c
$ git clone --depth=1 --branch=0.0.0-rc10 https://github.com/roma-glushko/tango.git roma-glushko/tango
Cloning into 'roma-glushko/tango'...
travis_time:end:0d1a397c:start=1568011483080530200,finish=1568011484843065800,duration=1762535600,event=checkout
$ cd roma-glushko/tango
$ git checkout -qf 0.0.0-rc10
travis_fold:end:git.checkout

travis_time:end:0d1a397c:start=1568011483080530200,finish=1568011485091393600,duration=2010863400,event=checkout
$ travis_export_go 1.12.x github.com/roma-glushko/tango
travis_time:start:106f5b02

Setting environment variables from .travis.yml
$ export PATH=/snap/bin:$PATH
$ export GO111MODULE=on

travis_time:end:106f5b02:start=1568011485185325200,finish=1568011485308841100,duration=123515900,event=env
travis_time:start:1a884014
$ travis_setup_go
Note: checking out 'e13541b555075c431e1215cac670f0f322540627'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

$ export GOPATH="/c/Users/travis/gopath"
$ export PATH="/c/Users/travis/gopath/bin:/c/Users/travis/.gimme/versions/go1.12.9.windows.amd64/bin:/snap/bin:/c/Users/travis/bin:/bin:/usr/bin:/c/tools/ruby25/bin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/ProgramData/GooGet:/c/Program Files/Google/Compute Engine/metadata_scripts:/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/c/Program Files/Google/Compute Engine/sysprep:/c/Program Files/Docker:/c/ProgramData/chocolatey/bin:/c/Program Files/CMake/bin:/c/Program Files/Git/cmd:/c/Program Files/LLVM/bin:/c/Program Files/dotnet:/c/Users/travis/AppData/Local/Microsoft/WindowsApps:/c/ProgramData/chocolatey/lib/mingw/tools/install/mingw64/bin"
$ export GO111MODULE="on"
go version go1.12.9 windows/amd64
travis_time:end:1a884014:start=1568011485359253600,finish=1568011518400310300,duration=33041056700,event=
$ gimme version
v1.5.3
$ go version
go version go1.12.9 windows/amd64
travis_fold:start:go.env
$ go env
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\travis\AppData\Local\go-build
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:/Users/travis/gopath
set GOPROXY=
set GORACE=
set GOROOT=C:\Users\travis\.gimme\versions\go1.12.9.windows.amd64
set GOTMPDIR=
set GOTOOLDIR=C:\Users\travis\.gimme\versions\go1.12.9.windows.amd64\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\travis\gopath\src\github.com\roma-glushko\tango\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\travis\AppData\Local\Temp\go-build318432194=/tmp/go-build -gno-record-gcc-switches
travis_fold:end:go.env
travis_fold:start:install
travis_time:start:0fbdf0c9
$ travis_install_go_dependencies 1.12.x -v
go get: -t flag is a no-op when using modules
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
get "golang.org/x/sys": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys?go-get=1
go: finding github.com/oschwald/geoip2-golang v1.3.0
go: finding github.com/urfave/cli v1.21.0
go: finding github.com/stretchr/testify v1.3.0
go: finding github.com/cheggaaa/pb v0.0.0-20190702094313-40231cf7fa00
go: finding github.com/oschwald/maxminddb-golang v1.3.1
go: finding golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7
Fetching https://gopkg.in/yaml.v2?go-get=1
go: finding github.com/BurntSushi/toml v0.3.1
Parsing meta tags from https://gopkg.in/yaml.v2?go-get=1 (status code 200)
get "gopkg.in/yaml.v2": found meta tag get.metaImport{Prefix:"gopkg.in/yaml.v2", VCS:"git", RepoRoot:"https://gopkg.in/yaml.v2"} at https://gopkg.in/yaml.v2?go-get=1
go: finding golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb
go: finding gopkg.in/yaml.v2 v2.2.2
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding github.com/davecgh/go-spew v1.1.0
go: finding github.com/fatih/color v1.7.0
go: finding github.com/mattn/go-colorable v0.1.2
go: finding github.com/mattn/go-runewidth v0.0.4
go: finding github.com/stretchr/objx v0.1.0
Fetching https://gopkg.in/check.v1?go-get=1
Parsing meta tags from https://gopkg.in/check.v1?go-get=1 (status code 200)
get "gopkg.in/check.v1": found meta tag get.metaImport{Prefix:"gopkg.in/check.v1", VCS:"git", RepoRoot:"https://gopkg.in/check.v1"} at https://gopkg.in/check.v1?go-get=1
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: finding github.com/mattn/go-isatty v0.0.8
go: finding golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223
go: downloading github.com/urfave/cli v1.21.0
go: downloading github.com/oschwald/geoip2-golang v1.3.0
go: downloading github.com/cheggaaa/pb v0.0.0-20190702094313-40231cf7fa00
go: extracting github.com/cheggaaa/pb v0.0.0-20190702094313-40231cf7fa00
go: downloading github.com/mattn/go-runewidth v0.0.4
go: extracting github.com/urfave/cli v1.21.0
go: extracting github.com/oschwald/geoip2-golang v1.3.0
go: downloading github.com/oschwald/maxminddb-golang v1.3.1
go: downloading github.com/BurntSushi/toml v0.3.1
go: downloading gopkg.in/yaml.v2 v2.2.2
go: extracting github.com/mattn/go-runewidth v0.0.4
go: extracting github.com/oschwald/maxminddb-golang v1.3.1
go: extracting github.com/BurntSushi/toml v0.3.1
go: downloading golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7
go: extracting gopkg.in/yaml.v2 v2.2.2
go: extracting golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7
tango/internal/usecase/config
github.com/urfave/cli
github.com/BurntSushi/toml
gopkg.in/yaml.v2
github.com/mattn/go-runewidth
github.com/cheggaaa/pb
tango/internal/domain/entity
github.com/urfave/cli/altsrc
tango/internal/usecase/mapper
tango/internal/usecase/processor
tango/internal/cli/factory
tango/internal/usecase
golang.org/x/sys/windows
tango/internal/infrastructure/reader
tango/internal/cli/component
tango/internal/usecase/filter
github.com/oschwald/maxminddb-golang
github.com/oschwald/geoip2-golang
tango/internal/usecase/report
tango/internal/infrastructure/writer
tango/internal/di
tango/internal/cli/command
tango/internal/cli
tango
travis_time:end:0fbdf0c9:start=1568011519825266700,finish=1568011549226993700,duration=29401727000,event=install
travis_fold:end:install
travis_fold:start:before_script
travis_time:start:0840730e
$ go get -t -v ./...
go get: -t flag is a no-op when using modules
travis_time:end:0840730e:start=1568011549281840200,finish=1568011550391542600,duration=1109702400,event=before_script
travis_fold:end:before_script
travis_time:start:01d1a41c
$ go build
travis_time:end:01d1a41c:start=1568011550442358800,finish=1568011553653541700,duration=3211182900,event=script
The command "go build" exited with 0.
travis_time:start:2dd6fcf2
$ go test
?   	tango	[no test files]
travis_time:end:2dd6fcf2:start=1568011553704821000,finish=1568011567879185200,duration=14174364200,event=script
The command "go test" exited with 0.

travis_fold:start:dpl_0
travis_time:start:2713b210
$ ruby -S gem install dpl
Successfully installed dpl-1.10.12
Parsing documentation for dpl-1.10.12
Installing ri documentation for dpl-1.10.12
Done installing documentation for dpl after 0 seconds
1 gem installed
travis_time:end:2713b210:start=1568011568571112900,finish=1568011571759897300,duration=3188784400,event=after_success
travis_fold:end:dpl_0
travis_time:start:24551688
Successfully installed dpl-script-1.10.12
Parsing documentation for dpl-script-1.10.12
Installing ri documentation for dpl-script-1.10.12
Done installing documentation for dpl-script after 0 seconds
1 gem installed
The system cannot find the path specified.
The system cannot find the path specified.
main: line 10: rev: command not found
main: line 12: rev: command not found
travis_fold:start:dpl.1
Installing deploy dependencies

travis_fold:end:dpl.1
travis_fold:start:dpl.2
Preparing deploy

travis_fold:end:dpl.2
travis_fold:start:dpl.3
Deploying application

travis_fold:end:dpl.3
failed to deploy
Script failed with status 127

Error:

Parsing documentation for dpl-script-1.10.12
Installing ri documentation for dpl-script-1.10.12
Done installing documentation for dpl-script after 0 seconds
1 gem installed
The system cannot find the path specified.
The system cannot find the path specified.
main: line 10: rev: command not found
main: line 12: rev: command not found
travis_fold:start:dpl.1

Finally, I was able to dedicate a bit of time to fix a few blocking issues:

• https://github.com/roma-glushko/tango/issues/49 - first of all, Homebrew has deprecated some directives in their taps which broke Tango installation on macOS. A few people reported that to me but I guess, unfortunately, many more got affected.
• https://github.com/roma-glushko/homebrew-tango/issues/2 - since the last release, Apple released a new M1 processor architecture. In order to include it into Tango buillds, I have updated the project Golang version from 1.13 to the latest 1.18.

Changeset

• updated the Tango Brew Tag to remove deprecated directives
• brought support for Apple M1 processors
• updated project golang version to 1.18
• moved from TravisCI to Github Actions
• updated the GoReleaser config file to support all release destinations just like before

Description

The first gif in the readme directly caught my attention. I've been staring at it for 30 sec, to wait for the "demo" to start. But instead, it just starts over. An animation of a progress bar isn't really helpful.

I think it would be more awesome if it could show the output. Provide a glimpse of what this project provides. What happens once the logs have been "read"?

Additional information

Description

It's good to have a way to specify a custom browser classification from the config file.

Expected behavior

It's possible to specify that SuperCustomChCrawler2.0 is a part of the Crawler group.

Benefits

Let users better control their browser reports

Description

It should be possible to analyze access logs in multiple threads to speed up the tool.

Benefits

Better user experience and less time to wait during huge log analysis

Description

It should be possible to read access logs not only from files but also from streams like

zcat access.log.gz | tango custom -r custom.csv

Benefits

It would be possible to analyze compressed logs on the fly without unarchiving them.

Description

Currently, Tango is only working with memory during log analysis. On the large scale projects, it may be an issue to analyze access logs as they Gb-sized.

Expected behavior

It's possible to use disk space to store information during access log analysis.

Benefits

Better support of large-scale projects with a demand to analyze huge access logs

Additional information

B+Tree can be used here.

Description

This may be helpful to support analyzing a group of logs at the same time to include their data to the same report.

Benefits

Streamline analyzing of big amount of access logs