Blackfire Player is a powerful Web Crawling, Web Testing, and Web Scraper application. It provides a nice DSL to crawl HTTP services, assert responses, and extract data from HTML/XML/JSON responses.

Related tags

Scraping player
Overview

Blackfire Player

Blackfire Player is a powerful Web Crawling, Web Testing, and Web Scraper application. It provides a nice DSL to crawl HTTP services, assert responses, and extract data from HTML/XML/JSON responses.

Some Blackfire Player use cases:

  • Crawl a website/API and check expectations -- aka Acceptance Tests;
  • Scrape a website/API and extract values;
  • Monitor a website;
  • Test code with unit test integration (PHPUnit, Behat, Codeception, ...);
  • Test code behavior from the outside thanks to the native Blackfire Profiler integration -- aka Unit Tests from the HTTP layer (tm).

Read more about how to download and use Blackfire Player.

Comments
  • Add variables into multiline strings

    Add variables into multiline strings

    Current state:

    Multiline strings allow to easily use JSON for the requests but there is no way to use variable to set same specific key.

    Idea:

    We can use similar syntax as in Twig: {{ variableName }}, to use variable inside multiline variable.

    Example:

    set login admin
    set password qwerty
    
    scenario Test 1
        visit url('/')
            body
            """
            {
                login: "{{login}}",
                password: "{{password}}"
            }
            """
    

    Will send body like:

    {
        login: "admin",
        password: "qwerty"
    }
    
    opened by scyzoryck 11
  • Reusing extracted data from previous step

    Reusing extracted data from previous step

    Scenario like this works as expected:

    scenario:
        steps:
            - visit: url('http://example.com')
              extract:
                 hrefs: css("a").extract("href")
            - visit: url(hrefs[0])
              extract:
                  title: css("title")
    

    Would be nice if it can iterate over results from previous step:

    scenario:
        steps:
            - visit: url('http://example.com')
              extract:
                 hrefs: css("a").extract("href")
            - visit: url(hrefs)
              extract:
                  title: css("title")
    

    This gives: PHP Warning: ltrim() expects parameter 1 to be string, array given in phar:///usr/local/bin/blackfire-player.phar/Player/Guzzle/RequestFactory.php on line 60. Is there any workaround?

    opened by umpirsky 9
  • Regex don't return first match

    Regex don't return first match

    Step reproduce

    scenario.bkf

    name "Visitor"
    
    endpoint "http://domain.local/"
    
    set login_body '{ "username": "[email protected]", "password": "admin" }'
    
    scenario
        name "Customer login"
    
        visit url('/rest/V1/integration/customer/token')
            method 'POST'
            header "Content-Type: application/json"
            header "Accept: application/json"
            body login_body
            set token regex('/\\w+/')
            expect status_code() == 200
        visit url('/rest/V1/carts/mine')
            method 'POST'
            header "Content-Type: application/json"
            header "Accept: application/json"
            header "Authorization: Bearer " ~ token
            header "Cookie: PHPSESSID=fki34dvcj80g7ct23m2u80mci9"
            body ''
            set cart_id body()
            expect status_code() == 200
    

    Run command

    blackfire-player run scenario.bkf --variable login_body='{ "username": "[email protected]", "password": "admin" }'
    

    Throw exception

    Screenshot from 2021-03-07 17-07-12

    Response for api /rest/V1/integration/customer/token

    Screenshot from 2021-03-07 17-07-49

    Thanks.

    opened by dylanngo95 5
  • Make the installation secure

    Make the installation secure

    Make the installation https://blackfire.io/docs/player/index secure.

    • it's not using https://
    • no signature check

    I suggest you look at the phive project and sign the releases so they can be installed via phive. If you have questions, I am happy to assist.

    opened by amenk 5
  • Authorization with custom token not correct

    Authorization with custom token not correct

    Validate authorization on header not correct

    My custom token like: 531f1b07c466e186a4e3e8193514cc03:wgvqn3h7hbmzyld4rr86i65rt50jje44

    The header like: Authorization: Bearer 531f1b07c466e186a4e3e8193514cc03:wgvqn3h7hbmzyld4rr86i65rt50jje44

    blackfire-player reponse auth like: Authorization: Bearer 531f1b07c466e186a4e3e8193514cc03

    I don't login to my API by scenario. Please help me for fix it ? Thanks.

    opened by tinhnd2 4
  • Not usable with composer

    Not usable with composer

    The alpha 3 seems not to be usable with composer, anymore.

    The path in the blackfire-player.php in /bin to the autoload.php of composer is not correct and the link to the bin php file in /vendor/bin/ is missing.

    Would be great if the player is still usable as a dependency loaded by composer in the future :-)

    opened by staelche 4
  • Laravel integration

    Laravel integration

    Trying to install this via Composer into Laravel 5.2 results in:

    [InvalidArgumentException] Could not find package blackfireioplayer at any version for your minimum-st ability (stable). Check the package spelling or your minimum-stability

    Are you able to get the Blackfire Player to a stable state inside Composer?

    opened by absalomedia 4
  • Dev requirements look weird to me

    Dev requirements look weird to me

    The console component is listed as a dev requirement, while it is required to have it in the phar (same for other dev requirements btw). This would make them normal requirements

    opened by stof 4
  • ValidateCommand: Allow scenario input via stdin.

    ValidateCommand: Allow scenario input via stdin.

    This is analogous to the "run" command, which also accepts the scenario on stdin. It allows running the validator inside a Docker container without doing funny stuff to get the data inside.

    opened by UlrichEckhardt 3
  • Cannot access to JSON keys named `@id`

    Cannot access to JSON keys named `@id`

    I try to add Blackfire Player to the demo of API Platform: https://github.com/api-platform/demo/pull/91 It looks like it's not possible to access JSON keys containing a special char like @ (used by JSON-LD).

    Example

    scenario
        visit url('/books')
            name "Post a new book"
            method "POST"
            header "accept: application/ld+json"
            header "content-type: application/ld+json"
            body
            """
            {
              "isbn": "978-2-918390-36-7",
              "title": "Code performance explained",
              "description": "Blackfire.io, the book",
              "author": "Fabien Potencier",
              "publicationDate": "today"
            }
            """
            expect status_code() == 201
            set id json("'@id'")
    
        visit url(id)
            expect status_code() == 200
    

    It looks valid according to JMESPath, but it returns an empty string with BlackFire player. However, accessing to hydra:member (the colon is also a special char) works.

    opened by dunglas 3
  • Multiline arguments

    Multiline arguments

    Hey. Did you think about adding possibility to create multiline arguments in scenarios? It would increase readability of bigger JSON bodies for sending POST / PUT requests. :)

    opened by scyzoryck 3
  • [Suggestion] Add debug mode in

    [Suggestion] Add debug mode in "Profile all requests"

    Use case : When you want to profile a login form (or other HTTP POST) from player in "Profile all requests".

    Feature : Add a option for enable debug mode.

    opened by Theosakamg 1
  • Issue when trying to use `with`, `assert`, and variables together

    Issue when trying to use `with`, `assert`, and variables together

    Hi!

    I'm trying to combine the usage of with and assert in order to write a single visit url() block which will be driven by configuration (from the with).

    Here is our .blackfire.yaml:

    scenarios: |
        #!blackfire-player
    
        scenario
            name "Performance regression"
    
            with slug, data in \
                { \
                    '/a': { name: 'A', maxWallTime: '1s' }, \
                    '/b': { name: 'B', maxWallTime: '2s' }, \
                    '/c': { name: 'C', maxWallTime: '3s' }
                }
    
                visit url(slug)
                    samples 3
                    warmup true
                    name "Checking performance for " ~ data['name']
                    expect status_code() == 200
                    assert main.wall_time < data["maxWallTime"] * var('time_coeff', 1)
                    assert percent(main.wall_time) < 10%
    

    The issue is caused by the usage of data["maxWallTime"]. I've tried different syntaxes but it still fails (with different error message tho):

    1. assert main.wall_time < data["maxWallTime"] * var('time_coeff', 1):
    Unexpected characters (maxWallTime"]) at line 3 (near "assertions: ["main.wall_time < data["maxWallTime"] * var('time_coeff', 1)"]") in the request configuration (config_yml)
    
    1. assert main.wall_time < data['maxWallTime'] * var('time_coeff', 1):
    Invalid configuration for path "tests._assertion_2.assertions.0.expression": Variable "data" is not valid around position 18 for expression `main.wall_time < data['maxWallTime'] * var('time_coeff', 1)`. Did you forget to prefix your variable with "main.", "metrics.", "runtime.", or "vars."?
    
    1. assert main.wall_time < data.maxWallTime * var('time_coeff', 1):
    Invalid configuration for path "tests._assertion_2.assertions.0.expression": Variable "data" is not valid around position 18 for expression `main.wall_time < data.maxWallTime * var('time_coeff', 1)`. Did you forget to prefix your variable with "main.", "metrics.", "runtime.", or "vars."?
    

    Does someone know how to make things working?

    Thanks!

    opened by Kocal 1
  • New feature - Add basic support for JUnit reports

    New feature - Add basic support for JUnit reports

    Hello! I would like to add some basic support for JUnit reports - that becomes a nice standard for the showing reports from tests in tools like gitlab or junit. Here is some MVP PR for the basic support to show basic information about failed or passed tests. Supported features:

    • amount of executed scenarios (test suites) and steps (test cases)
    • failure and error messages in report Result has been checked agains Jenkins schema.

    Features that can be added in the future:

    • better naming - right now it shows raw scenario names - without resolved variables.
    • add time to the results
    opened by scyzoryck 1
  • Add self-update command

    Add self-update command

    Updating the player is a bit of a hassle as we need to actually download and install a new version over the previous one.

    Having a blackfire-player selfupdate would be very handy.

    enhancement 
    opened by lolautruche 1
  • assert inline multiline response

    assert inline multiline response

    the docs at https://blackfire.io/docs/player/index#visiting-a-page-with-visit mention a request notation using a long inline string:

    scenario
        visit url('/login')
            method 'POST'
            body
            """
            {
                "user": "john",
                "password": "doe"
            }
            """
    

    we tried using the very same notation for asserting the response body, but were not able to get a result without a syntax error within the config file.

    could you describe a small example on how to expect body() using a long multiline response body string?

    we tried things like

    scenario
        visit url('/login')
            method 'POST'
            body
            """
            {
                "user": "john",
                "password": "doe"
            }
            """
            expect status_code() == 200
            expect body
            """
            {
                "status": "success"
            }
            """
    

    and also tried

    scenario
        visit url('/login')
            method 'POST'
            body
            """
            {
                "user": "john",
                "password": "doe"
            }
            """
            expect status_code() == 200
            expect body()
            """
            {
                "status": "success"
            }
            """
    

    .. both dont work

    opened by staabm 2
  • Blackfire player error

    Blackfire player error

    hi,

    im trying to install blackfire-player by following instructions on this section:

    https://github.com/blackfireio/player#download

    and when im trying to run it im getting this error:

    [XXX@XXXX-dev1 ~]$ blackfire-player PHP Parse error: syntax error, unexpected '(' in phar:///path/to/blackfire-player.phar/Player/Console/PlayerCommand.php on line 70

    thanks!

    opened by gioCastillo 3
Releases(v1.22.0)
Owner
Blackfire
Fire up your Application Performance
Blackfire
Crawlzone is a fast asynchronous internet crawling framework aiming to provide open source web scraping and testing solution.

Crawlzone is a fast asynchronous internet crawling framework aiming to provide open source web scraping and testing solution. It can be used for a wide range of purposes, from extracting and indexing structured data to monitoring and automated testing. Available for PHP 7.3, 7.4, 8.0.

null 68 Dec 27, 2022
This script scrapes the HTML from different web pages to get the information from the video and you can use it in your own video player.

XVideos PornHub RedTube API This script scrapes the HTML from different web pages to get the information from the video and you can use it in your own

null 57 Dec 16, 2022
Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc.

Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc.

AOE 0 Sep 14, 2021
A small example of crawling another website and extracting the required information from it to save the website wherever we need it

A small example of crawling another website and extracting the required information from it to save the website wherever we need it Description This s

Mohammad Qasemi 9 Sep 24, 2022
Library for Rapid (Web) Crawler and Scraper Development

Library for Rapid (Web) Crawler and Scraper Development This package provides kind of a framework and a lot of ready to use, so-called steps, that you

crwlr.software 60 Nov 30, 2022
Goutte, a simple PHP Web Scraper

Goutte, a simple PHP Web Scraper Goutte is a screen scraping and web crawling library for PHP. Goutte provides a nice API to crawl websites and extrac

null 9.1k Jan 1, 2023
PHP Scraper - an highly opinionated web-interface for PHP

PHP Scraper An opinionated & limited way to scrape the web using PHP. The main goal is to get stuff done instead of getting distracted with xPath sele

Peter Thaleikis 327 Dec 30, 2022
Goutte, a simple PHP Web Scraper

Goutte, a simple PHP Web Scraper Goutte is a screen scraping and web crawling library for PHP. Goutte provides a nice API to crawl websites and extrac

null 9.1k Jan 4, 2023
The most integrated web scraper package for Laravel.

Laravel Scavenger The most integrated web scraper package for Laravel. Top Features Scavenger provides the following features and more out-the-box. Ea

Reliq Arts 134 Jan 4, 2023
PHP scraper for ZEE5 Live Streaming URL's Using The Channel ID and Direct Play Anywhere

It can scrape ZEE5 Live Streaming URL's Using The Channel ID and Direct Play Anywhere

null 1 Mar 24, 2022
Extractor (scraper, crawler, parser) of products from Allegro

Extractor (scraper, crawler, parser) of products from Allegro

Daniel Yatsura 1 May 11, 2022
Simple and fast HTML parser

DiDOM README на русском DiDOM - simple and fast HTML parser. Contents Installation Quick start Creating new document Search for elements Verify if ele

null 2.1k Dec 30, 2022
A configurable and extensible PHP web spider

Note on backwards compatibility break: since v0.5.0, Symfony EventDispatcher v3 is no longer supported and PHP Spider requires v4 or v5. If you are st

Matthijs van den Bos 1.3k Dec 28, 2022
A program to scrape online web-content (APIs, RSS Feeds, or Websites) and notify if search term was hit.

s3n Search-Scan-Save-Notify A program to scrape online web-content (APIs, RSS Feeds, or Websites) and notify if search term was hit. It is based on PH

Aamer 11 Nov 8, 2022
Get info from any web service or page

Embed PHP library to get information from any web page (using oembed, opengraph, twitter-cards, scrapping the html, etc). It's compatible with any web

Oscar Otero 1.9k Jan 1, 2023
Property page web scrapper

Property page web scrapper This tool was built to expermiment with extracting features for property pages on websites like booking.com and Airbnb. Thi

Vaugen Wake 2 Feb 24, 2022
Roach is a complete web scraping toolkit for PHP

?? Roach A complete web scraping toolkit for PHP About Roach is a complete web scraping toolkit for PHP. It is heavily inspired (read: a shameless clo

Roach PHP 1.1k Jan 3, 2023
It can Scrap ZEE5 Live Streaming URL's Using The Channel ID and Direct Play Anywhere

It can Scrap ZEE5 Live Streaming URL's Using The Channel ID and Direct Play Anywhere

Techie Sneh 21 Nov 19, 2021
Assert - Assertions to validate method input/output with nice error messages.

Webmozart Assert This library contains efficient assertions to test the input and output of your methods. With these assertions, you can greatly reduc

Webmozarts GmbH 7.2k Dec 30, 2022