Please make sure all of your classes have proper namespaces so that they may be safely used in a 3rd-party project. I would highly recommend adhering to the PSR-4 autoloading standard: http://www.php-fig.org/psr/psr-4/

In this changeset I've done two changes, whereas one may be disputed. First I've removed all regex functions. This gave a very small improvement, almost negligible, but it's something.

Secondly, I made the file_position method optional on the listener. This is probably a more controversial change, but the performance benefit is 15-20%. If it was removed entirely, the performance benefit would be between 20-25%. Unfortunately, if you actually rely on file_position, this change set has a performance penalty compared to master. For me this is a no-brainer, but assume others will have some opinions.

Put differently: Parsing a 500MB file with ~1.2M entries, I now save 10 minutes.

I did 10 test runs (for run in {1..10}; do php tests/performance.php; done) of the perf branch vs master, here are the results:

| phaza/perf | salsify/master | gain | | --- | --- | --- | | 9.107668877 | 10.67672706 | 14.70% | | 8.819965124 | 11.25982618 | 21.67% | | 9.094146967 | 10.73865008 | 15.31% | | 8.585276127 | 10.50739408 | 18.29% | | 9.22768712 | 10.99455309 | 16.07% | | 9.032989979 | 10.81993604 | 16.52% | | 8.707053185 | 10.64168501 | 18.18% | | 9.159993887 | 10.44938397 | 12.34% | | 8.928499937 | 10.909729 | 18.16% | | 8.877639771 | 11.15229297 | 20.40% | | ----------------- | ------------------ | ---------- | | 8.954092097 | 10.81501775 | 17.16% |

I might be missing something here, but it looks like the listener class can't call the parser's stop(), because it doesn't have a parser object. The perfect place to introduce them would be in Parser::__construct(), where both objects are ready for use.

setParser(Parser $parser) would be optimal, but that would break bc for classes implementing the interface directly.

What I did to 'fix' this, is throw and handle a ParsingStoppedException, but that seems silly since there is a stop() method...

I know that this is a stupid question, but how to install the JsonStreamingParser via composer? I've tried with "php-streaming-json-parser": "4.*" and "JsonStreamingParser" and what not..

To reproduce the behavior, modify example.json to :

[
  {
    "name": "example document for wicked fast parsing of huge json docs",
    "integer": 123,
    "totally sweet scientific notation": -123.123e-2,
    "unicode? you betcha!": "ú™£¢∞§\u2665",
    "zero character": "0",
    "null is boring": null,
    //this is the important part
    "lines": [
      { "test" : 1, "test2" : 2},
      { "test" : 1, "test2" : 2}
    ]
  },
  ...
]

Using the example.php script, it'll produce:

  [0]=>
  array(7) {
    ...
    [0]=> //this key should be "list" not "0"
    array(2) {
      [0]=>
      array(2) {
        ["test"]=>
        int(1)
        ["test2"]=>
        int(2)
      }
      [1]=>
      array(2) {
        ["test"]=>
        int(1)
        ["test2"]=>
        int(2)
      }
    }
  }

I first though it was because of the start_object method but it's not. I've not looked into the core script to see if I could fix this but if someone had a clue on that it'd be great! Is it a bug or is this the indented behavior?

I m read trying to 36 MB File Fails,its taking too much time. $str = file_get_contents('./JString.json'); $arr = json_decode($str, true); // decode the JSON into an associative array

Fast too much than given code jsonstreamingparser...........

Entirely up to the project owner, but it helps when reasoning about dependencies and maintenance if you know that a given dependency adheres to a particular versioning standard. The current tags show v1.0, v2.0, v3.0, v4.0 but I'm not sure if they represent breaking changes or just updates. I'm also keen to use the Name\Spaced\Update on master but not sure if master is really stable...

http://semver.org/

Can you provide better examples on how to use this? PHP still returns memory issues:

Allowed memory size of 134217728 bytes exhausted

The examples are lacking. All it shows is a way to read the entire file into memory, which is what I thought this code is supposed to be avoiding.

Currently jsonstreamingparser is not a ready-to-use solution. It provides the following readers (listeners):

• GeoJsonListener.php - reads GeoJSON
• IdleListener.php - reads nothing
• InMemoryListener.php - reads all in memory
• SubsetConsumerListener.php - didn't get what it does as no description was provided.

--- neither of which can be used with a arbitrary JSON data. Maybe that's so because the author didn't find any really useful and generic reader (except for GeoJSON which is certainly not generic)? That could be the case, yeah, and if we look at the project README, we would clearly read the message *simply implement your Listener and it's done!". That easy!

Is that? Well guys I've tried to implement a so called Listener and I can tell you: IT'S BLOODY DIFFICULT, cumbersome and time consuming. Honestly I don't see any point in implementing something more difficult then just a reader of objects going one after another. Because if you really want to preprocess json, you'd definitely use a specialized tool like jq.

So yeah, the generic case exists and it's called: reader of top-level objects.

By a good fortune it is already implemented here: https://github.com/MAXakaWIZARD/JsonCollectionParser

My proposition is to include it in the list of "standard" readers (listeners).

Also, let's:

• consider not including its Parser as it doesn't seem to be generic;
• finish the work started here: #60 (add support for a stream of json documents);
• finish the work started here (the same support for document streams, also objects, objects in subarrays etc)

Let's summon @MAXakaWIZARD for the discussion.

Hi,

I am consuming some third-party JSON which does not escape \x7F byte in strings.

src/Parser.php#L218 does consider this invalid. But i think other libraries do not consider this invalid.

Is there any more information about this issue?

When we import a UTF8 file fread starts with reading the BOM signature for the file and it causes an error in parsing the stream with error Document must start with object or array.

I believe the problem is fully answered here and should be a fairly easy fix :smile:

https://stackoverflow.com/questions/9126423/php-fread-function-returning-extra-charactors-at-the-front-on-utf-8-text-files

Thanks in advance

Hi, thank you for this awesome tool, it was extremely helpful parsing huge GeoJSON data. However, the explanation of the really important part is totally missing. It was difficult to find out what's going on. When I went to your blog post the same happened, when I was looking for how to implement the listener the blog post just say: It's pretty straightforward. It is but not for somebody who gets here the first time :)

Here's my code snippet using the GeoJSON Listener for another users who needs a better example:

<?php

  ini_set('max_execution_time', 0);
  ini_set('memory_limit', -1);

  $tmp_file_name = 'geojson.json';

  $stream = fopen('/tmp/'.$tmp_file_name, 'r');

  $listener = new \JsonStreamingParser\Listener\GeoJsonListener(function($json){
      if(!empty($json)){
          //var_dump($json['properties']);
          ////////////////// Do WHATEVER YOU WANT WITH THE FEATURE //////////////////

          //Storing in db
          $data = [
              'title'     => !empty($json['properties']['shapeName']) ? $json['properties']['shapeName'] :$json['properties']['shapeISO'],
              'iso'     => $json['properties']['shapeISO'],
              'gb_id'     => $json['properties']['shapeID'],
              'country_iso'     => $json['properties']['shapeGroup'],
              'admin_level'     => str_replace('ADM', '', $json['properties']['shapeType']),
          ];

          $json['properties'] = $data;

          $data['geometry'] = json_encode($json);

          try{
               //Mock DB insert
              DB::table('geoboundaries')->insert($data);
          }catch (\Exception $exception){
              dd(substr($exception->getMessage(), 0, 300));
          }



      }

  });
  try {
      $parser = new \JsonStreamingParser\Parser($stream, $listener);
      $parser->parse();

      fclose($stream);
      return true;
  } catch (Exception $e) {
      fclose($stream);
      throw $e;
  }

So I'm trying to write a wrapper over this lib that would be used as a generator. Basically I want to yield all json-objects I need from a stream.

The lib does not use yields or generators, so this is what I came up with:

First, a simple listener for this example: lets collect all values from "foo" fields given they all are scalar

class FooJsonListener extends \JsonStreamingParser\Listener\IdleListener {
    protected $isReadingFoo = false;
    protected $onFooCallback;
    
    public function onFoo(callable $onFooCallback)
    {
        $this->onFooCallback = $onFooCallback;
    }

    public function key(string $key): void
    {
        $this->isReadingFoo = ($key === 'foo');
    }

    public function value($value): void
    {
        if ($this->isReadingFoo) {
            call_user_func($this->onFooCallback, $value);
            $this->isReadingFoo = false;
        }
    }
}

Now this function starts parser, then stops every time we collected a foo-value, yields it, then resumes parsing again:

function fooIterator($stream) {
    $jsonListener = new FooJsonListener();
    $jsonParser = new JsonStreamingParser\Parser($stream, $jsonListener);

    $lastFoo = null;
    $shouldYieldFoo = false;
    $jsonListener->onFoo(function ($foo) use (&$lastFoo, &$shouldYieldFoo, $jsonParser) {
        $lastFoo = $foo;
        $shouldYieldFoo = true;
        $jsonParser->stop();
    });

    while (true) {
        $jsonParser->parse();
        if ($shouldYieldFoo) {
            yield $lastFoo;
            $shouldYieldFoo = false;
        } else {
            break;
        }
    }
}

Looks awful, but should work. But it does not. After the first yield, next parse() will throw:

JsonStreamingParser\Exception\ParsingException: "Parsing error in [1:1]. Expected ',' or ']' while parsing array. Got: {"

It works if we don't interrupt it with "stop()" thought.

The problem: parse() reads stream by chunks, but parses char-by-char. On any char there might be a call to listener to register parsed token. After every char it checks stopParsing flag, raised by stop(). So most of the cases, when you call stop() from listener you break chunk parsing in the middle, leftover of the chunk is discarded, and calling parse() then will proceed from the next chunk, not where it stopped. And also stopParsing is never reset to false.

Possible fixes: a. Store the actual count of bytes read on stop(), and do fseek to this offset on next parse() (would not work with non-seekable streams thought) b. Better: store the leftover of chunk on stop() and prepend it to data read on next parse()

Sadly, neither of this is possible to implement by extending classes due to the privates.

When installing this code with compser for Magento 2 it creates a symlink to phpunit at vendor/bin/phpunit. But since phpunit is not required in the composer.json it doesn't get isntalled. Because of this it creates a broken symlink for Magento to.

I get this error after installing the library with composer:

PHP Fatal error:  Uncaught Error: Class 'JsonStreamingParser\Listener\InMemoryListener' not found in /data/www/default/database/json_to_sql.php:14
Stack trace:
#0 {main}
  thrown in /data/www/default/database/json_to_sql.php on line 14

json_to_sql.php

require_once 'vendor/autoload.php';
$json_file = dirname(__FILE__) . "file.json";

# Load json data from the file into a variable
Line 14 -> $listener = new \JsonStreamingParser\Listener\InMemoryListener();

$stream = fopen($json_file, 'r');
try {
    $parser = new \JsonStreamingParser\Parser($stream, $listener);
    $parser->parse();
    fclose($stream);
} catch (Exception $e) {
    fclose($stream);
    throw $e;
}

$json_data = $listener->getJson();
var_dump($json_data);

I've installed the library with downloading the composer.json file and doing php composer.phar install.

$ php composer.phar diagnose                                              
Checking composer.json: OK
Checking platform settings: OK
Checking git settings: 

  [Symfony\Component\Process\Exception\RuntimeException]
  The Process class relies on proc_open, which is not available on your PHP installation.

diagnose

When I clone the repository it gives me the same error. I don't know if I have to do an include or something, there is no example.

The parser doesn't give an error when the input suddenly stops. The current behavior results in "nothing more to process" and you think you are done. When it happens halfway in one of the blocks, that block simply "disappear".

This is easily to reproduce with several unit tests by cutting of a bit at the end of the test files. If you simply remove the "}" at the end of the file, the tests still work. The tests only start to fail when you remove the "}" at level 2.

The Parser should throw an exception when the json is not correctly closed, as this might be an indication you are missing part of a (potentially big) file.