An Elasticsearch engine plugin for Moodle's Global Search

Overview

Build Status

Moodle Global Search - Elasticsearch Backend

This plugin allows Moodle to use Elasticsearch as the search engine for Moodle's Global Search.

The following features are provided by this plugin:

  • Multiple versions of Elasticsearch
  • File indexing
  • Request signing, compatible with Amazon Web Services (AWS)
  • Respects Moodle Proxy settings
  • Image recognition and indexing (via external service)

Supported Moodle Versions

This plugin currently supports Moodle:

Moodle version Branch
Moodle 3.10 to 3.11 MOODLE_310_STABLE
Moodle 3.5 to 3.9 master

Elasticsearch Version Support

Currently this plugin is tested to work against the following versions of Elasticsearch:

  • 5.5.0
  • 6.0.0
  • 6.4.1
  • 6.6.1

Verified Platforms

This plugin has been tested to work on the following cloud platforms:

Generic Elasticsearch Setup

To use this plugin first you will need to setup an Elaticsearch service.

The following is the bare minimum to get Elasticsearch working in a Debian/Ubuntu Operating System environment. Consult the Elasticsearch Documention for in depth instructions, or for details on how to install on other operating systems.

NOTE: The instructions below should only be used for test and dev purposes. Don't do this in production. For a production setup we recommend Elasticsearch running as a cluster, getting started documentation can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html

Elasticsearch requires Java as a prerequisite, to install Java:

sudo apt-get install default-jre default-jdk

Once Java is installed, the following commands will install and start Elasticsearch.

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.1.deb
sudo dpkg -i elasticsearch-6.6.1.deb
sudo update-rc.d elasticsearch defaults
sudo service elasticsearch start

A quick test can be performed by running the following from the command line.

curl -X GET 'http://localhost:9200'

The output should look something like:

{
  "name" : "1QHLiux",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "mLRqIsnVRrGdgg2OfHWNrg",
  "version" : {
    "number" : "5.1.2",
    "build_hash" : "c8c4c16",
    "build_date" : "2017-01-11T20:18:39.146Z",
    "build_snapshot" : false,
    "lucene_version" : "6.3.0"
  },
  "tagline" : "You Know, for Search"
}

By default the Elasticsearch service is available on: http://localhost:9200

Azure Elasticsearch Setup

To use this plugin first you will need to setup an Elaticsearch service.

To use Microsoft Azure to provide an Elasticsearch service for Moodle:

  1. Create a Microsoft Azure account: Account creation page
  2. Create a Linux virtual machine and connect to virtual machine: Azure Linux virtual machine setup guide
  3. Setup an Elasticsearch service:

Elasticsearch requires Java as a prerequisite, to install Java:


sudo apt-get install default-jre default-jdk

Once Java is installed, the following commands will install and start Elasticsearch.


wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.deb
sudo dpkg -i elasticsearch-5.5.0.deb
sudo update-rc.d elasticsearch defaults
sudo service elasticsearch start

A quick test can be performed by running the following from the command line.


curl -X GET 'http://localhost:9200'

The output should look something like:


{
  "name" : "1QHLiux",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "mLRqIsnVRrGdgg2OfHWNrg",
  "version" : {
    "number" : "5.1.2",
    "build_hash" : "c8c4c16",
    "build_date" : "2017-01-11T20:18:39.146Z",
    "build_snapshot" : false,
    "lucene_version" : "6.3.0"
  },
  "tagline" : "You Know, for Search"
}

AWS Elasticsearch Setup

To use this plugin first you will need to setup an Elaticsearch service.

To use Amazon Webservices (AWS) to provide an Elasticsearch service for Moodle:

  1. Create an AWS account: Account creation guide
  2. Setup an Elasticsearch service: AWS Elasticsearch setup guide

Moodle Plugin Installation

Once you have setup an Elasticsearch service you can now install the Moodle plugin.

These setup steps are the same regardless of how you have setup the Elasticsearch service.

  1. Get the code and copy/ install it to: <moodledir>/search/engine/elastic
  2. This plugin also depends on local_aws get the code from https://github.com/catalyst/moodle-local_aws and copy/ install it into <moodledir>/local/aws (This is required regardless of how your Elasticsearch service is being supplied.)
  3. Run the upgrade: sudo -u www-data php admin/cli/upgrade Note: the user may be different to www-data on your system.

Moodle Plugin Setup

Once you have setup an Elasticsearch service you can now configure the Moodle plugin.

These setup steps are the same regardless of how you have setup the Elasticsearch service.

  1. Log into Moodle as an administrator
  2. Set up the plugin in Site administration > Plugins > Search > Manage global search by selecting elastic as the search engine.
  3. Configure the Elasticsearch plugin at: Site administration > Plugins > Search > Elastic
  4. Set hostname and port of your Elasticsearch server
  5. Optionally, change the Request size variable. Generally this can be left as is. Some Elasticsearch providers such as AWS have a limit on how big the HTTP payload can be. Therefore we limit it to a size in bytes.
  6. To create the index and populate Elasticsearch with your site's data, run this CLI script. sudo -u www-data php search/cli/indexer.php --force
  7. Enable Global search in Site administration > Advanced features

File Indexing Support

This plugin uses Apache Tika for file indexing support. Tika parses files, extracts the text, and return it via a REST API.

Tika Setup

Seting up a Tika test service is straight forward. In most cases on a Linux environment, you can simply download the Java JAR then run the service.


wget http://apache.mirror.amaze.com.au/tika/tika-server-1.16.jar
java -jar tika-server-1.16.jar

This will start Tika on the host. By default the Tika service is available on: http://localhost:9998

Enabling File indexing support in Moodle

Once a Tika service is available the Elasticsearch plugin in Moodle needs to be configured for file indexing support.
Assuming you have already followed the basic installation steps, to enable file indexing support:

  1. Configure the Elasticsearch plugin at: Site administration > Plugins > Search > Elastic
  2. Select the Enable file indexing checkbox.
  3. Set Tika hostname and Tika port of your Tika service. If you followed the basic Tika setup instructions the defaults should not need changing.
  4. Click the Save Changes button.

What is Tika

From the Apache Tika website:

The Apache Tika™ toolkit detects and extracts metadata and text from over a thousand different file types (such as PPT, XLS, and PDF). All of these file types can be parsed through a single interface, making Tika useful for search engine indexing, content analysis, translation, and much more. You can find the latest release on the download page. Please see the Getting Started page for more information on how to start using Tika.

Why use Tika as a stand alone service?

It is common to see Elasticsearch implementations using an Elasticsearch file indexing plugin rather than a stand alone service. Current Elasticsearch plugins are a wrapper around Tika. (The Solr search engine also uses Tika).
Using Tika as a standalone service has the following advantages:

  • Can support file indexing for Elasticsearch setups that don't support file indexing plugins such as AWS.
  • No need to change setup or plugins based on Elasticsearch version.
  • You can share one Tika service across multiple Elasticsearch clusters.
  • Can run Tika on dedicated infrastructure that is not part of your search nodes.
  • Files stored using native Elasticsearch functionality are stored as separate records inside Elasticsearch, these are separate to the rest of the data stored relating to that file.
  • Ingesting files using native Elasticsearch functionality is very inefficient. Files are stored in the Elasticsearch internal database as base64 encoded strings. Base64 on average takes up 30% more space than the original binary. This is in addition to the content extracted from the file which is also stored in Elasticsearch.
  • The Elasticsearch documentation also states:
Extracting contents from binary data is a resource intensive operation and consumes a lot of resources. It is highly recommended to run pipelines using this processor in a dedicated ingest node.

Image Recognition and Indexing

This plugin can use the Amazon Web Services (AWS) [Rekognition service(https://aws.amazon.com/rekognition/) to identify the contents of images. The identified content is then indexed by Elasticsearch and can be searched for in Moodle (cool huh?).

NOTE: Indexing of files by Moodle's core Global Search is currently limited to only indexing files from a couple of places. Tracker issue MDL-59459 has been raised to increase the coverage of the files indexed by Global Search.

Currently the best resource to use to test image search functionality it so add an image via the Moodle course file resource.

Enabling image recognition and indexing support in Moodle

Once you have setup Elasticsearch in AWS Moodle needs to be configured for Image Recognition.
Assuming you have already followed the basic installation steps and the file indexing steps, to enable Image Recognition:

  1. Configure the Elasticsearch plugin at: Site administration > Plugins > Search > Elastic
  2. Select the Enable image signing checkbox.
  3. Set Key ID, Secret Key and Region of your AWS credentials and Rekognition region.
  4. Click the Save Changes button.

NOTE: You will need a set of AWS API keys for an AWS IAM user with full Rekognition permissions. Setting this up is beyond the scope of this README. for further information see the AWS Documentation.

Request Signing

Amazon Web Services (AWS) provide Elasticsearch as a managed service. This makes it easy to provision and manage and Elasticsearch cluster.
One of the ways you can secure access to your data in Elasticsearch when using AWS is to use request signing. Request signing allows only valid signed requests to be accepted by the Elasticsearch endpoint. Requests that are unsigned are not authorised to access the endpoint.

Enabling Request Signing support in Moodle

Once you have setup Elasticsearch in AWS Moodle needs to be configured for Request Signing.
Assuming you have already followed the basic installation steps, to enable Request Signing:

  1. Configure the Elasticsearch plugin at: Site administration > Plugins > Search > Elastic
  2. Select the Enable request signing checkbox.
  3. Set Key ID, Secret Key and Region of your AWS credentials and Elasticsearch region.
  4. Click the Save Changes button.

Webservices

This plugin exposes two AJAX enabled webservices, to allow you to integrate Moodle's Global search with other systems and services. The two available webservices are:

  • search_elastic_search - Returns search results based on provided search query.
  • search_elastic_search_areas - Returns the search area IDs for each available search area in Moodle.

Setup and documentation of these services is connsistent with other Moodle core web services.

This plugin sets up a pre-configured External service called Search service when the plugin is installed. This service adds and enables the two webservice methods provided by this plugin.

NOTE: You will need to have Global search and this plugin enabled and configured correctly before you can use the provided web services.

Test Setup

In order to run the PHP Unit tests for this plugin you need to setup and configure an Elasticsearch instance as will as supply the instance details to Moodle. You need to define:

  • Hostname: the name URL of the host of your Elasticsearch Instance
  • Port: The TCP port the host is listening on
  • Index: The name of the index to use during tests. NOTE: Make sure this is different from your production index!

Setup via config.php

To define the required variables in via your Moodle configuration file, add the following to config.php:


define('TEST_SEARCH_ELASTIC_HOSTNAME', 'http://127.0.0.1');
define('TEST_SEARCH_ELASTIC_PORT', 9200);
define('TEST_SEARCH_ELASTIC_INDEX', 'moodle_test_2');

Setup via Environment variables

The required Elasticserach instance configuration variables can also be provided as environment variables. To do this at the Linux command line:


export TEST_SEARCH_ELASTIC_HOSTNAME=http://127.0.0.1; export TEST_SEARCH_ELASTIC_PORT=9200; export TEST_SEARCH_ELASTIC_INDEX=moodle_test

Running the tests

First initialise the test environment, from the Moodle code home directory: php admin/tool/phpunit/cli/init.php To run only this plugins tests: vendor/bin/phpunit search_elastic_engine_testcase search/engine/elastic/tests/engine_test.php

Crafted by Catalyst IT

This plugin was developed by Catalyst IT Australia:

https://www.catalyst-au.net/

Catalyst IT

Contributing and Support

Issues, and pull requests using github are welcome and encouraged!

https://github.com/catalyst/moodle-search_elastic/issues

If you would like commercial support or would like to sponsor additional improvements to this plugin please contact us:

https://www.catalyst-au.net/contact-us

Comments
  • File indexing crashes if file is missing in Moodle filedir

    File indexing crashes if file is missing in Moodle filedir

    Hi Matt,

    I am currently looking into search_elastic and have added the latest version of this plugin to a Moodle 3.2.3+ (Build: 20170622) instance and have hooked this up to a fresh elasticsearch 5.5 instance.

    I also started a standalone tika instance running on a separate machine and configured this tika instance in the plugin's settings.

    While doing the first indexing with sudo -u apache /opt/rh/rh-php70/root/usr/bin/php /var/www/html/moodle_dev3/search/cli/indexer.php --force with fileindexing enabled, the indexer script encountered a fatal error and stopped with this message:

    ////Default exception handler: Die Datei kann nicht gelesen werden. Eventuell existiert sie nicht oder es gibt ein Rechteproblem. Debug: [dataroot]/filedir/c2/40/c24091a092e5afc7310088ce2e416d1d7efcda11
    Error code: storedfilecannotread
    * line 579 of /lib/filestorage/stored_file.php: file_exception thrown
    * line 274 of /search/engine/elastic/classes/document.php: call to stored_file->get_imageinfo()
    * line 353 of /search/engine/elastic/classes/engine.php: call to search_elastic\document->export_file_for_engine()
    * line 511 of /search/engine/elastic/classes/engine.php: call to search_elastic\engine->process_document_files()
    * line 588 of /search/classes/manager.php: call to search_elastic\engine->add_document()
    * line 75 of /search/cli/indexer.php: call to core_search\manager->index()
    
    !!! Die Datei kann nicht gelesen werden. Eventuell existiert sie nicht oder es gibt ein Rechteproblem. !!!
    !! [dataroot]/filedir/c2/40/c24091a092e5afc7310088ce2e416d1d7efcda11
    Error code: storedfilecannotread !!
    !! Stack trace: * line 579 of /lib/filestorage/stored_file.php: file_exception thrown
    * line 274 of /search/engine/elastic/classes/document.php: call to stored_file-&gt;get_imageinfo()
    * line 353 of /search/engine/elastic/classes/engine.php: call to search_elastic\document-&gt;export_file_for_engine()
    * line 511 of /search/engine/elastic/classes/engine.php: call to search_elastic\engine-&gt;process_document_files()
    * line 588 of /search/classes/manager.php: call to search_elastic\engine-&gt;add_document()
    * line 75 of /search/cli/indexer.php: call to core_search\manager-&gt;index()
     !!
    
    

    The german sentences in the debug message mean that the file couldn't be read from disk. This conclusion is correct as I ran the indexing on a Moodle test instance which was rsynced from our production system with excluding big files in Moodledata (almost always videos) for storage saving reasons. Unfortunately, due to this fact, I am unable to test indexing properly on this test instance.

    Would it be possible to check if a file really exists on disk before it is sent to the file indexing backend and ignore it otherwise?

    Thanks, Alex

    opened by abias 6
  • Support for Moodle 3.5

    Support for Moodle 3.5

    One issue in classes/query.php, line 321: For example, this code is never executed because it seams that $usercontents is an object in 3.5 and not an array // Add contexts. if (gettype($usercontexts) == 'array') { $contexts = $this->construct_contexts($usercontexts); array_push ($query['query']['bool']['filter']['bool']['must'], $contexts); } Option "Search within enrolled courses only" does not have any effect than

    Another change I had to make to make it work is this (added groupid, line 118): $excludedfields = array('itemid', 'areaid', 'courseid', 'contextid', 'userid', 'owneruserid', 'modified', 'type', 'groupid' // added group id, otherwise it always fails when searching for string );

    opened by mglumac 5
  • Enable file indexing retroactively is not possible

    Enable file indexing retroactively is not possible

    Steps to reproduce:

    • Don't enable the "Enable file indexing" setting in search_elastic
    • Fully index your Moodle instance
    • Enable the "Enable file indexing" setting in search_elastic retroactively

    Expected result:

    • Moodle / search_elastic will index the existing files additionally to the existing content

    Actual result:

    • There isn't anything indexed additionally

    I assume that this additional index can't be made automatically under the hood and that it basically needs a full re-index of the existing content to also index the existing files. That's why I would just propose to add this fact to the description of the "Enable file indexing" setting in search_elastic.

    bug Confirmed! 
    opened by abias 5
  • Support for Tika as an ElasticSearch plugin?

    Support for Tika as an ElasticSearch plugin?

    Hi,

    I am currently looking into search_elastic to run it as an alternativ to search_solr, mainly because Elasticsearch seems to be easier to install and run on our RHEL 7 systems.

    I have seen that you recommend running Tika for file indexing as a standalone application (see https://github.com/catalyst/moodle-search_elastic#tika-setup). However, there are no rpm packages for Tika out there as far as I see and fiddling with a manually configured service for Tika can be daunting.

    On the other hand, you also write that there are Elasticsearch plugins for Tika. I have found https://www.elastic.co/guide/en/elasticsearch/plugins/current/ingest-attachment.html and would like to ask:

    • Is this plugin the Tika plugin you are mentioning?
    • Is search_elastic able to work with Tika as an Elasticsearch plugin or does it really need Tika as a standalone application?

    Thanks, Alex

    enhancement wontfix 
    opened by abias 5
  • Cannot connect to standalone tika instance

    Cannot connect to standalone tika instance

    Hi Matt,

    I am currently looking into search_elastic and have added the latest version of this plugin to a Moodle 3.2.3+ (Build: 20170622) instance and have hooked this up to a fresh elasticsearch 5.5 instance.

    I also started a standalone tika instance running on a separate machine and configured this tika instance in the plugin's settings.

    While doing the first indexing with sudo -u apache /opt/rh/rh-php70/root/usr/bin/php /var/www/html/moodle_dev3/search/cli/indexer.php --force with fileindexing enabled, the indexer script encountered a fatal error and stopped with this message:

    PHP Notice:  Undefined variable: client in /var/www/html/moodle_dev3/search/engine/elastic/classes/document.php on line 157
    
    Notice: Undefined variable: client in /var/www/html/moodle_dev3/search/engine/elastic/classes/document.php on line 157
    Default exception handler: Fehler: Call to a member function post() on null Debug: 
    Error code: generalexceptionmessage
    * line 157 of /search/engine/elastic/classes/document.php: Error thrown
    * line 286 of /search/engine/elastic/classes/document.php: call to search_elastic\document->extract_text()
    * line 352 of /search/engine/elastic/classes/engine.php: call to search_elastic\document->export_file_for_engine()
    * line 510 of /search/engine/elastic/classes/engine.php: call to search_elastic\engine->process_document_files()
    * line 588 of /search/classes/manager.php: call to search_elastic\engine->add_document()
    * line 75 of /search/cli/indexer.php: call to core_search\manager->index()
    
    !!! Fehler: Call to a member function post() on null !!!
    !! 
    Error code: generalexceptionmessage !!
    !! Stack trace: * line 157 of /search/engine/elastic/classes/document.php: Error thrown
    * line 286 of /search/engine/elastic/classes/document.php: call to search_elastic\document-&gt;extract_text()
    * line 352 of /search/engine/elastic/classes/engine.php: call to search_elastic\document-&gt;export_file_for_engine()
    * line 510 of /search/engine/elastic/classes/engine.php: call to search_elastic\engine-&gt;process_document_files()
    * line 588 of /search/classes/manager.php: call to search_elastic\engine-&gt;add_document()
    * line 75 of /search/cli/indexer.php: call to core_search\manager-&gt;index()
     !!
    

    I traced the problem back to commit 4c32c7103f9b1579807e02a743cfedf292247df2 which breaks the connection to tika. Based on the latest code, this patch should solve the problem and clean up the function at the same time:

    diff --git a/classes/document.php b/classes/document.php
    index 6a574df..50ab93b 100644
    --- a/classes/document.php
    +++ b/classes/document.php
    @@ -148,19 +148,18 @@ class document extends \core_search\document {
          */
         private function extract_text($file) {
             // TODO: add timeout and retries for tika.
    -        $config = get_config('search_elastic');
             $extractedtext = '';
             $port = $this->tikaport;
    -        $hostname = rtrim($this->tikahostname, "/");
    +        $hostname = $this->tikahostname;
             $url = $hostname . ':'. $port . '/tika/form';
     
    +        $client = new \curl();
             $response = $client->post($url, array('file' => $file));
             if ($client->info['http_code'] === 200) {
                 $extractedtext = $response;
             }
     
             return $extractedtext;
    -
         }
     
         /**
    

    However, I am wondering how this problem could remain undetected as you are running this plugin in production...

    Thanks, Alex

    bug Confirmed! 
    opened by abias 4
  • Add subplugin architecture

    Add subplugin architecture

    Refactor plugin to use a subplugin architecture. The Tika text extraction and AWS Rekognition Image rekognition features should be sub plugins. This would make it easier to manage and create integrations to other services. For example to use Google's image recognition service instead of AWS

    enhancement wontfix 
    opened by mattporritt 4
  • Search keyword is order dependant

    Search keyword is order dependant

    eg searching for "dentist lithograph" shows a result, but searching for "lithograph dentist" doesn't find the same result. Further more searching for "lithograph OR dentist" doesn't return anything either.

    https://s-cqu-mba.catalyst-au.net/search/?page=0&q=dentist%20lithograph&title&timestart=0&timeend=0

    bug Confirmed! 
    opened by brendanheywood 4
  • Document Mapping Failing

    Document Mapping Failing

    Currently there is an issue when the document mapping is created for the index. This is causing all document field types to be set to "text" instead of integer, date etc. This is causing date sorting to not work for search results. It is also causing strange result behaviour.

    Also to make it worse, this condition happens on a real site, but does not happen for unit tests. The unit tests get the document created correctly.

    Real site document mapping

    curl -XGET 'http://localhost:9200/moodle2/_mapping?pretty=true' 
    {
      "moodle2" : {
        "mappings" : {
          "doc" : {
            "properties" : {
              "areaid" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "content" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "contextid" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "courseid" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "description1" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "id" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "itemid" : {
                "type" : "long"
              },
              "modified" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "owneruserid" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "parentid" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "title" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              },
              "type" : {
                "type" : "long"
              },
              "userid" : {
                "type" : "long"
              }
            }
          }
        }
      }
    }
    

    Unit test document mapping

    curl -XGET 'http://localhost:9200/moodle_test/_mapping?pretty=true' 
    {
      "moodle_test" : {
        "mappings" : {
          "doc" : {
            "properties" : {
              "areaid" : {
                "type" : "keyword"
              },
              "content" : {
                "type" : "text"
              },
              "contextid" : {
                "type" : "integer"
              },
              "courseid" : {
                "type" : "integer"
              },
              "id" : {
                "type" : "keyword"
              },
              "itemid" : {
                "type" : "integer"
              },
              "modified" : {
                "type" : "date",
                "format" : "epoch_second"
              },
              "owneruserid" : {
                "type" : "integer"
              },
              "parentid" : {
                "type" : "keyword"
              },
              "title" : {
                "type" : "text"
              },
              "type" : {
                "type" : "integer"
              }
            }
          }
        }
      }
    }
    
    bug Confirmed! 
    opened by mattporritt 3
  • Debug messages with indexing the

    Debug messages with indexing the "Messages - received" and "Messages - sent" area

    Hi Matt,

    I am currently looking into search_elastic and have added the latest version of this plugin to a Moodle 3.2.3+ (Build: 20170622) instance and have hooked this up to a fresh elasticsearch 5.5 instance.

    While doing the first indexing with sudo -u apache /opt/rh/rh-php70/root/usr/bin/php /var/www/html/moodle_dev3/search/cli/indexer.php --force I saw that there are tons of CLI debug messages for the "Messages - received" and "Messages - sent" area telling me:

    ++ Error retrieving core_message-message_sent 2111873 document, not all required data is available: Ungültige Nutzer/in ++
    * line 59 of /message/classes/search/base_message.php: call to debugging()
    * line 61 of /message/classes/search/message_sent.php: call to core_message\search\base_message->get_document()
    * line ? of unknownfile: call to core_message\search\message_sent->get_document()
    * line 103 of /lib/classes/dml/recordset_walk.php: call to call_user_func()
    * line 573 of /search/classes/manager.php: call to core\dml\recordset_walk->current()
    * line 75 of /search/cli/indexer.php: call to core_search\manager->index()
    

    ("Ungültige Nutzer/in" is the german term for "Invalid user" as I have set $CFG->lang = 'de' in config.php.

    However, after some time and a very long CLI output, the indexing job comes to an end.

    I had also quickly setup a SOLR instance some weeks ago and I can't remember that indexing the same Moodle instance with SOLR had also thrown these kind of errors.

    The only reason for these problems I can think of is that we are using auth_ldap sync on a regular basis to delete Moodle accounts which have disappeared in LDAP, so there might be messages in the Moodle database which don't have a connected sender or receiver Moodle account anymore.

    In the end, I am wondering if these debug messages come from your plugin or from Moodle core and if I should worry about them or not.

    Thanks in advance, Alex

    invalid 
    opened by abias 3
  • Problems with setup on Moodle 3.4

    Problems with setup on Moodle 3.4

    There is an error in the setup in Moodle 3.4 and 3.5 where the plugin returns "Section Error" and we cna get no further in the installation. ( we down graded and the plugin and it worked fine)

    to3dmoq

    And this is the error: 5ref1a6

    opened by neeesn 2
  • Requirement for local_aws on a local ElasticSearch instance?

    Requirement for local_aws on a local ElasticSearch instance?

    Hi,

    I am currently looking into search_elastic to run it as an alternativ to search_solr, mainly because Elasticsearch seems to be easier to install and run on our RHEL 7 systems.

    I have seen that search_elastic requires local_aws. May I ask if local_aws is really necessary for local ElasticSearch instances (i.e. not in AWS)?

    If it is really necessary, could you please also publish local_aws to the Moodle plugins repository so that we can install it from there and get update notifications?

    If it is not necessary, could you please remove this requirement from version.php and e.g. replace it with some custom checks if local_aws is installed before a AWS instance can be configured?

    Thanks, Alex

    opened by abias 2
  • Test support for 4.1 -> Some issues found

    Test support for 4.1 -> Some issues found

    I have tested installation, automated tests, and basic indexing/searching on Moodle 4.1. I found the following minor issues:

    • Unit tests need renaming and namespacing to match current conventions.
    • Some tests use assertions which are deprecated in newer versions of PHPUnit.
    • Tests for proxy configuration fail if a proxy is configured for the development environment.
    opened by marxjohnson 0
  • Modernise the way proxy connections work

    Modernise the way proxy connections work

    This will redo the work done in this issue:

    https://github.com/catalyst/moodle-search_elastic/issues/9

    and do it so that all the proxy logic is handled centrally in local_aws instead

    https://github.com/catalyst/moodle-local_aws/blob/master/classes/local/client_factory.php#L44-L60

    opened by brendanheywood 1
  • Any plans on supporting Elasticsearch 8.x?

    Any plans on supporting Elasticsearch 8.x?

    The existing code does not seem to work as-is with Elasticsearch 8.x, due to removal of types. This is causing any endpoint URL with /doc/ to return 400 - Bad Request. Also, the creation of index from the plugin fails because "doc" is no longer supported in create index BODY:

    {
      "mappings": {
        "doc": {
          "properties": {
    ...
          }
        }
      }
    }'
    

    https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html

    Steps to replicate: Install Elasticsearch 8.2. https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html Try adding/updating index from Moodle.

    opened by pnavadia 4
Releases(2020101200)
Owner
Catalyst IT
We specialise in developing, designing and supporting enterprise grade systems using open source technologies.
Catalyst IT
Nova Search is an open source search engine developed by the Artado Project.

Loli Search Loli Search açık kaynak kodlu bir arama motorudur ve yalnızca kendi sonuçlarını değil, diğer arama motorlarının sonuçlarını da göstermekte

Artado Project 10 Jul 22, 2022
Doogle is a search engine and web crawler which can search indexed websites and images

Doogle Doogle is a search engine and web crawler which can search indexed websites and images, and then use keywords to be searched later. Written pri

Zepher Ashe 9 Jan 1, 2023
🪃 Zero-dependency global `kirbylog()` helper for any content

Kirbylog The most simple, Kirby-esque way to log content to file. Most of the time, I just want to log some string or array to a file. That's what thi

Johann Schopplich 11 Nov 9, 2022
Fixes WordPress 5.9 global CSS styles specificity issues

Fixes WordPress 5.9 global CSS styles specificity issues

Oliver Juhas 3 Feb 22, 2022
Packet-driven global form interaction-spam blocker for PM.

Looking for testers and README improvers! Form Interaction Fix Interaction-spam Interaction spam is often a problem for players who use the mouse as t

EndermanbugZJFC 5 Dec 16, 2022
search non profitable charity or organization through api search

Non Profile Charity Search Search non profitable organization or get the details of an organization Installation Require the package using composer: c

Touhidur Rahman 5 Jan 20, 2022
Silverstripe-fulltextsearch - Adds external full text search engine support to SilverStripe

FullTextSearch module Adds support for fulltext search engines like Sphinx and Solr to SilverStripe CMS. Compatible with PHP 7.2 Important notes when

Silverstripe CMS 42 Dec 30, 2022
Magento 2 Module for Search Engine Optimization

Magento 2 Search Engine Optimization Magento 2 Module to Improve Search Engine Optimization (SEO) on your Magento site. Installation Install the modul

Stämpfli AG 100 Oct 7, 2022
A privacy respecting free as in freedom meta search engine for Google and popular torrent sites

A privacy respecting free as in freedom meta search engine for Google and popular torrent sites

null 329 Dec 27, 2022
Magento ElasticSearch Implementation

Magento 2.x support : The repo for Magento 2 version of our module is available in the new repository of the module : https://github.com/Smile-SA/elas

Smile - Open Source Solutions 153 Nov 21, 2022
A plugin manager for PocketMine-MP downloads plugin from PocketMine-MP official plugin repository

oh-my-pmmp A plugin manager for PocketMine-MP Getting Started Prerequisites Your server MUST RUN the latest version of PocketMine. Installation From P

thebigcrafter 6 Jan 4, 2023
Integrates the ClassicPress Plugin Directory and any plugin stored in GitHub (tagged with classicpress-plugin) in the ClassicPress Admin

ClassicPress Plugin Directory Adds a new screen in a ClassicPress Install where you can browse, install, activate, deactivate, update, delete and pagi

TukuToi 3 Dec 27, 2022
Shiki is a beautiful syntax highlighter powered by the same language engine that many code editors use.

Shiki is a beautiful syntax highlighter powered by the same language engine that many code editors use. This package allows you to use Shiki from PHP.

Spatie 229 Jan 4, 2023
Engine for performing and rendering text diffs

Text_Diff Engine for performing and rendering text diffs This package provides a text-based diff engine and renderers for multiple diff output formats

PEAR - PHP Extension and Application Repository 15 Jan 4, 2022
Simple JQL builder for Jira search

Jql Builder Simple JQL builder for Jira search Installation composer require devmoath/jql-builder Usage Generate query with one condition: \DevMoath\J

Moath 11 Jan 2, 2023
Open App Engine

Open App Engine

Light App Engine 20 Dec 18, 2022
Production-grade rapid controller development with built in love for API and Search

Installation For CakePHP 4.x compatible version: composer require friendsofcake/crud For CakePHP 3.x compatible version: composer require friendsofca

Friends Of Cake 357 Jan 2, 2023
Lumen on Docker - Skeleton project with Nginx, MySQL & PHP 8 | Aws ECS, Google Kubernates, Azure Container Engine

Docker infrastructure for Lumen Description Microservice Lumen is a starting skeleton based on Docker and Lumen Framework. This project helps to devel

Fabrizio Cafolla 218 Sep 25, 2022
Google Search Results PHP API via Serp Api

Google Search Results in PHP This Php API is meant to scrape and parse Google, Bing or Baidu results using SerpApi. The full documentation is availabl

SerpApi 42 Nov 14, 2022