AbuseIO is a toolkit to receive, process, correlate and notify about abuse reports received by network operators, typically hosting and access providers.

I have a new install of AbuseIO via Ansible and it seems that Analytics screen is not functioning. None of the options are clickable nor is there a way to enter a date range or do other filtering. I have tried in Chrome and Safari.

While doing migration from 4.0 to 4.1 , I followed steps given in https://docs.abuse.io/en/latest/migration/.

4.0 to 4.1 Migration was successful.

After migration found that found that additional table "ticket_graph_points" added in 4.1, which has missing analytics data.

How to make use of "ticket_graph_points" table for existing 4.0 DB data ?

https://github.com/AbuseIO/parser-google

Hi everyone,

Just wondering if anyone else has the same problem as I do, constant failing google abuse reports (SBR) failing to be parsed.

I'm doing some troubleshooting now but I think there may be a few possible source:

• DKIM parsing problems which causes the parsing to fail (In which case you may be able to make use of something such as https://github.com/angrychimp/php-dkim which can parse the DKIM or build something to do so if this is the problem )
• Multiple "Received:" Headers?

If anyone's experienced this previously or has any ideas I'd love some input!

Dank je 😄

Installation done successfully with PHP 5.6.32.

Abuseio version4.0.0, Found that framework.log was not created in /var/log/abuseio.

Also able to see tickets when imported sample mails, however when tried to create ticket by sending email to [email protected] not created any ticket.

Nothing in postfix log of abuseio server. from sender's logs it shows message delivered

Could you help what goes wrong here?

Unable to get the ticket creation part to work. The server is able to receive emails however any email being sent to notifier@domain doesn't create a ticket in the system.

None of the log files present in /var/log/abuseio show any information related to this. PostFix logs show the email was received.

I've seen that some mail systems (Microsoft 365, for one) seem to truncate the attachments on emails that get bounded to admins. These are the emails which AbuseIO has not been able to parse, and so it send to be triaged manually.

I've therefore added support for bundling attachments in a .zip file instead of attaching them as-is to the bounce email. From my testing, this stops the attachments from getting truncated.

I've added the new config option app.attachment_format (APP_ATTACHMENT_FORMAT environment variable) to enable overriding the default (I've named multifile) with the new option zipfile).

If no config changes are made, things carry on as-is. If attachment_format is changed to zipfile, it will create a .zip in /tmp, add all attachments into it, then send that to the admin instead.

This is with default notification configuration.

Exception trace: () at /opt/abuseio/bootstrap/cache/compiled.php:14081 Illuminate\Foundation\Bootstrap\HandleExceptions->handleError() at /opt/abuseio/bootstrap/cache/compiled.php:14081 Illuminate\View\Factory->make() at /opt/abuseio/vendor/laravel/framework/src/Illuminate/Foundation/helpers.php:748 view() at /opt/abuseio/vendor/abuseio/notification-mail/src/Mail.php:130 AbuseIO\Notification\Mail->send() at /opt/abuseio/app/Jobs/Notification.php:44 AbuseIO\Jobs\Notification->send() at /opt/abuseio/app/Jobs/Notification.php:111 AbuseIO\Jobs\Notification->walkList() at /opt/abuseio/app/Console/Commands/Housekeeper/NotificationsCommand.php:141 AbuseIO\Console\Commands\Housekeeper\NotificationsCommand->handle() at n/a:n/a call_user_func_array() at /opt/abuseio/bootstrap/cache/compiled.php:1187 Illuminate\Container\Container->call() at /opt/abuseio/vendor/laravel/framework/src/Illuminate/Console/Command.php:150 Illuminate\Console\Command->execute() at /opt/abuseio/vendor/symfony/console/Command/Command.php:256 Symfony\Component\Console\Command\Command->run() at /opt/abuseio/vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command->run() at /opt/abuseio/vendor/symfony/console/Application.php:844 Symfony\Component\Console\Application->doRunCommand() at /opt/abuseio/vendor/symfony/console/Application.php:189 Symfony\Component\Console\Application->doRun() at /opt/abuseio/vendor/symfony/console/Application.php:120 Symfony\Component\Console\Application->run() at /opt/abuseio/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:107 Illuminate\Foundation\Console\Kernel->handle() at /opt/abuseio/artisan:36

Note that the line number for Mail.php is not the right one since I added some debug code to try and figure this out. The problematic line is:

$htmlmail = view(['template' => $htmlmail], $replacements)->render();

What does this PR do?

I have added a feature that enables admins to ignore incidents for certain IPs/Subnets in a netblock.

Aren't we able to do this anyway?

It is currently possible but one would have to split a netblock in two, in order to ignore a single IPv4, a solution that is not convenient (especially for several, sparse IP addresses in a netblock).

So can't they just mark the created tickets as ignored?

That would suffice as well, but it becomes irritating for our customers' sysadmins.

How do we test this?

All you have to do is drop the config file (config/whitelist.json) under your config folder (development or production based on your APP_ENV in .env), add 'whitelist' to your 'interface/navigation' bar in main.php. Then you can test it under /admin/whitelist. You can update the list with the following values:

• A single IPv4 ,e.g. 147.102.10.10
• An IPv4 CIDR subnet, e.g. 147.102.10.0/24

After you are done updating the whitelist press Update. At this point, you can feed the pipeline with emails in order to test that the right incidents are being ignored.

Does this feature add any dependencies?

No

UI View

Motivation

We (GRNET) are using AbuseIO as we are an ISP for several public entities (e.g. academic institutions, hospitals, etc.). Recently a research team from a lab under our constituency deployed several high-interaction honeypots for research purposes. That triggered our AbuseIO source (internal and external), and so several tickets opened. We were asked to stop sending tickets for specific IPv4 addresses. Unfortunately, there was no straightforward/scalable way to do that.

Awaiting feedback! :grinning:

Looking at report that come out of coloclue's abuse.io server, I see the timezone is missing:

Without timezone specification, the date is meaningless.

I recommend ISO 8601, example: 2016-10-15T13:09:16Z is used.

Ticket #305: Report for IP address XXX (Open Netbios Server)
Category: Information message, we strongly advice this matter to be resolved
Report date: 2016-08-28 01:37
Report count: 1
Source: Shadowserver
Reply or help: https://ash.abuseio.coloclue.net/?id=305&token=XX
Report information:
  - mac_address: 00-00-00-00-00-00
  - workgroup: WORKGROUP
  - machine_name: SERVER
  - username: SERVER
  - Address: XXX

Hi

I have a problem of understanding and hope that someone can help. According to the documentation it should be possible to receive mails instead of your own MTA by accessing a mailbox (IMAP or POP3?) and parse them.

I just don't understand how I can make this happen. There are no instructions for this, I have overlooked or misunderstood this feature? I'm using AbuseIO 4.0.3.

Can someone help me with that? How can I give AbuseIO access to an existing mailbox (which has been set up on the mail server for this purpose)? Which file (module) is responsible for this und wie kann ich dieses aktivieren?

Thank you in advance.

PS: What does this option/URL mean (as it is not available in my installation)?

'ash' => [ 'url' => 'https://abuseio.isp.local/ash/', ],

Best regards, Joh

If I have a comma and a whitespace as address separators in a contact, for example: "[email protected], [email protected]", the following error occurs when parsing a new email ticket:

Jun 7 18:26:50 machine abuseio[30191]: production.INFO: AbuseIO\Parsers\Shadowserver: Parser run completed for module : Shadowserver Jun 7 18:26:50 machine abuseio[30191]: production.INFO: AbuseIO\Parsers\Shadowserver: : Parser completed with 0 warnings and collected 26 incidents to save Jun 7 18:26:51 machine abuseio[30191]: production.INFO: AbuseIO\Jobs\IncidentsValidate: Validator has ended without errors Jun 7 18:26:51 machine abuseio[30191]: production.ERROR: AbuseIO\Jobs\IncidentsSave: Saver has ended with errors ! : DevError: Internal validation failed when saving the Ticket object All ip contact email addresses should be valid. Jun 7 18:26:51 machine abuseio[30191]: production.ERROR: AbuseIO\Jobs\EmailProcess: Email processor ending with errors. The received e-mail will be deleted from archive and bounced to the admin for investigation

It works when I don't have whitespaces in the list of addresses: "[email protected],[email protected]"

This could be fixed in /opt/abuseio/app/Providers/ValidationsServiceProvider.php, by trimming whitespaces in the "validation for multiple comma separated e-mails" code.

Since a few months ago, Netcraft stopped sending a "report.txt" attachment, and instead now sends an "xarf.json" attachment (see anonymised examples).

The parser-netcraft now bounces these messages, instead of parsing them. If you require any other samples to fix this, please let me know.

xarf-spam.json.txt xarf-webshell.json.txt xarf-malware.json.txt

Hey, I'm trying to install AbuseIO on my Ubuntu 22.04 Server. After fillign th database credentials to the .env-File I un into this error during php artisan migrate...

https://paste.robin-it.group/fedakobeve.rb

It would be useful to allow setting some configuration options through environment variables, instead of requiring to change the respective values directly through the source code. All current changes simply allow setting the respective options through environment variables, but keep the default options intact for compatibility purposes.

For now just FYI: I haven't looked at the specifics of how this will effect the shadowserver parser:

Dear Shadowserver Subscriber,

This is an important update on the upcoming changes to the Shadowserver sinkhole and honeypot report types and formats. We have recently updated our free public benefit reports service by introducing changes to our sinkhole, honeypot and darknet report types. These changes include introducing new report names, types and more standardized formats. These changes will affect how you parse and process our data. The new report types and formats are currently simultaneously delivered to you with the old reports that they are meant to replace.

Please make yourself (and all your mailing list contacts) familiar with the new report changes that are in effect and published on our website via URL, along with the rationale behind these changes:

https://www.shadowserver.org/news/changes-in-sinkhole-and-honeypot-report-types-and-formats/

Please note, as an existing subscriber, you will continue to receive the old reports that are being changed until 2021-06-01, as well as a copy of the new reports that we have introduced.

If you have any questions or concerns please contact us. We hope that these changes introduce cleaner and more future proof formats, which will allow for a better understanding of our reports and for easier and more flexible processing of our data.

Thanks

The Shadowserver Foundation

Error upon install using composer (version 2.0.11):

Creating a "abuseio/abuseio" project at "./abuseio"
Installing abuseio/abuseio (v4.3.0)
  - Downloading abuseio/abuseio (v4.3.0)
  - Installing abuseio/abuseio (v4.3.0): Extracting archive
Created project in /opt/abuseio
Loading composer repositories with package information
Updating dependencies
...
...
Generating optimized autoload files
Class tests\Api\Account\ApiVersionTest located in ./tests/Api/ApiVersionTest.php does not comply with psr-4 autoloading standard. Skipping.
Class Wpb\String_Blade_Compiler\Facades\StringBlade located in ./vendor/wpb/string-blade-compiler/src/Facade/StringBlade.php does not comply with psr-4 autoloading standard. Skipping.
> Illuminate\Foundation\ComposerScripts::postAutoloadDump
> @php artisan package:discover

In PackageManifest.php line 122:

  Undefined index: name

Script @php artisan package:discover handling the post-autoload-dump event returned with error code 1

@kruisdraad

It appears that filters for both ipechelon and copyrightnotice parsers are broken.

Let's look at ipechelon, as copyright notice appears to share a lot of code with it.

Firstly, the following filters are set in the config:

https://github.com/AbuseIO/parser-ipechelon/blob/4cfc1b9e1472e6c98d93df7556d4cedccbdb9fe3/config/Ipechelon.php#L28-L32

Primo, all of these are outside of $report_raw['Source'] which we apply filters to

https://github.com/AbuseIO/parser-ipechelon/blob/4cfc1b9e1472e6c98d93df7556d4cedccbdb9fe3/src/Ipechelon.php#L87

And then we finally save the $report_raw, completely ignoring $report.

https://github.com/AbuseIO/parser-ipechelon/blob/4cfc1b9e1472e6c98d93df7556d4cedccbdb9fe3/src/Ipechelon.php#L98

I am happy to fix these but want to understand what we want to collect for the incident.

Here is an example of what is currently saved, but the use of $report_raw['Source'] above is quite confusing, so I am not sure what was the intention here. Can you clarify what you've intended to capture and what should be ignored?

Case ID: [redacted]
Case Status: Open
Case Severity: Normal
Complainant Entity: Paramount Pictures Corporation
Complainant Contact: IP-Echelon - Compliance
Complainant Address: 6715 Hollywood Blvd Los Angeles CA 90028 United States of America
Complainant Phone: +1 (310) 606 2747
Complainant Email: p2p@[redacted]
Service_Provider Entity: [redacted]
Service_Provider Email: abuse@[redacted]
Source TimeStamp: 2020-03-12T19:02:00Z
Source IP_Address: [redacted]
Source Port: 54599
Source Type: BitTorrent
Source SubType @attributes: This is filtered due to fourth layer nesting
Source Number_Files: 1
Content Item TimeStamp: 2020-03-12T19:02:00Z
Content Item Title: Sonic the Hedgehog
Content Item FileName: Sonic The Hedgehog (2020) [1080p] [WEBRip] [YTS.MX]
Content Item FileSize: 1772912495
Content Item Hash: f3acfd3979cc1a30cc7f312673ced688ce78ce77