I've been working on a 3.6 site that needs support for Solr 5. Therefore, I've branched off the 2.4.0 tag and added support for it. The changes are almost entirely in the solrconfig.xml and types.ss file and were created from a copy of the templates/4 folder. This work carried on from some work undertaken by @zarocknz and @firesphere helped with some implementation details.

List of changes:

• solrconfig.xml: luceneMatchVersion 6.0
• solrconfig.xml: JsonUpdateRequestHandler -> UpdateRequestHandler
• solrconfig.xml: CsvUpdateRequestHandler -> UpdateRequestHandler
• solrconfig.xml: remove node containing requestHandler name="/admin"
• types.ss: remove all instances of enablePositionIncrements="true"
• Solr.php: enable checks for version 5
• Solr5Service: extended classes from Solr4Service

I've kept the "default" version of the module at 4 to avoid making this a breaking change, but if a user wants to use 5, they'll need to specify that in their config (example provided in docs)

I'll raise a separate pull request to enable SS4 support when I get the time to do so.

The module makes no assumption about what criteria might be applied for records to be added to the index. I'm proposing elsewhere that we filter out records with anonymous canView=false.

In addition, I think we should add behavior to prevent indexing of a record based on custom, model specific flags like ShowInSearch. Currently, the docs mention this usage pattern (e.g. https://www.cwp.govt.nz/developer-docs/en/2/features/solr_search/default_search_index/), and recommend adding this as a filter field. You're still in charge of filtering on that field by default. The CWPSearchEngine performs this filter out of the box, but only on SiteTree and File (see pull request). If you're defining your own index on other objects, or include other objects in that index, you need to create your own search engine - the filter field is indexed, but it's ignored during searches. The docs for searching dataobjects fail to mention that as well. This illustrates why we shouldn't have those records in the index in the first place.

This has some of the same rollout issues as filter out records with anonymous canView=false, we'd need to advise devs on how to get those existing index entries with ShowInSearch=false out of their index.

When releasing this, we should strongly recommend to search users that they run a reindex to prune any records with ShowInSearch=0 in their current search state.

Another thing to check: when we change ShowInSearch from 1 to 0 and click save/publish, does it get removed from the index straight away?

ACs

• [x] Records which have a ShowInSearch method or database property returning false are not indexed
• [x] This applies to built-in records (Page, File) as well as custom record definitions
• [x] This behaviour applies to both CWP and core module search
• [x] The ShowInSearch value is inspected every time a record is saved (non-versioned) or published (versioned), and the index entry is updated accordingly
• [x] Existing index entries with ShowInSearch=0 can be pruned through a reindex
• [x] There is documentation both for CWP and generic module upgrade paths
• [x] Documentation is adjusted to state that you MUST filter out ShowInSearch during search request for safety (rather than reindexing) Steve: users don't need to filter out during search, happens automatically
• [x] Funtionality for filtering ShowInSearch during search requests is retained in order to minimise risk, but marked as deprecated and non-functional - Steve: have instead deleted functionality from cwp-search module

Note:

We still have to index the ShowInSearch property, otherwise search request filtering on existing implementations will fail

PRs:

https://github.com/silverstripe/silverstripe-fulltextsearch/pull/275 https://github.com/silverstripe/cwp/pull/260 https://github.com/silverstripe/cwp-search/pull/31

Subtasks

• [x] Work out central place to add $this->addFilterField('ShowInSearch') so that it's available in core
• [x] Investigate where solr index is being updated so that we can also delete from there
• [x] Test also works for getShowInSearch() (it does, comments below)
• [x] Investigate filtering solr outlet on DataObject.ShowInSearch (for sites that have not reindexed)
• [x] Determine if $obj->getShowInSearch() is relevant for outlet filtering (it is)
• [x] (Ingo) Investigate what it would take to index a field across models with the same name, so ShowInSearch rather than MyModel_ShowInSearch - Steve: there's no need for this as we'll be filtering all these results out before they get in the index anyway
• [x] (Ingo) add a warning label about creating non-deterministic getShowInSearch() implementations
• [x] Ensure inlet filtering applied to both cms save (dirty items) as well as solr_reindex
• [x] Ensure variantStateExcluded (draft?) functionality still works for all cases above
• [x] Get agreement with Cheddam about a combined approach with the canView() checks
• [x] Test with frameworktest records to ensure that reindex still functions OK
• [x] Update documentation
• [x] Write unit tests for 2x inlet filtering, determined to be not feasible for outlet filtering
• [x] Write release comms saying run solr_reindex, possibly commit straight to changelog
• [x] Investigate deleting $this->addFilterField('ShowInSearch') and related code used in outlet filtering - possibly a separate optional PR

Links

https://silverstripeltd.slack.com/archives/CLXKD9X51/p1584670614347800

If you create a sub-class of SolrIndex to define indexes on DataObjects that don't use the Versioned extension, the index will be built and loaded into Solr, but no searches performed against that index will work.

For example:

// Story.php
class Story extends DataObject {
    private static $db = array(
        'Title' => 'Varchar(255)'
    );
}

// StorySolrIndex.php
class StorySolrIndex extends SolrIndex {
    public function init() {
        $this->addClass('Story');
        $this->addFulltextField('Title');
    }
}

Running Solr_Configure and Solr_Reindex will work fine (Solr_Reindex will say class Story, total: 5 in state [])

However, creating an instance of that SolrIndex and running a query against it will fail:

$index = new StorySolrIndex();
$query = new SearchQuery();
$query->search('Test', null, array(
    'Story_Title' => 2
));

$results = $index->search($query);

Nothing will be returned, but Solr will log an Exception with the message: SEVERE: org.apache.solr.common.SolrException: undefined field _versionedstage

If you add something to the StorySolrIndex that is versioned (e.g. SiteTree), then the search will work as usual, but with extra stuff in the index.

// StorySolrIndex.php
class StorySolrIndex extends SolrIndex {
    public function init() {
        $this->addClass('Story');
        $this->addClass('SiteTree'); // This works around the bug
        $this->addFulltextField('Title');
    }
}

The module denies access to a search result if you can’t view the underlying record: https://github.com/silverstripe/silverstripe-fulltextsearch/blob/f89818909a0d0c08f7d8f05217eb62e32988aa20/src/Solr/Forms/SearchForm.php#L89. It does however index all records, incl. ones with canView=false. The canView check is implemented in SearchForm->getResults(), which is a commonly subclassed method - so still a residual risk that devs will skip that check.

I think we should default to not indexing records where canView() returns false for anonymous users. You could overwrite this behaviour, e.g. to include silverstripe/secureassets in 3.x, or protected/private files in 4.x. But that would be a conscious decision, with the module staying secure by default.

We'd need to do this in a backwards compatible way (not breaking existing searches which rely on the absence of these checks). Although you could argue that those implementations rely on a bug?

When releasing this, we should strongly recommend to search users that they run a reindex to prune any records with canView=false in their current search state.

ACs

• [x] Records returning canView() as false for anonymous users are not indexed by default
• [x] This behaviour applies to draft pages and other versioned records in draft stage
• [x] This behaviour applies to draft files, as well as protected files
• [x] A reindex prunes out any records where canView() returns false that would've been previously indexed
• [x] When a new record is saved (unversioned) or published (versioned) in a way that changes anonymous canView() permissions, that's reflected by adding to or removing from the index
  • [x] This does not cover traversing child objects and reindexing them
• [x] The "exclude variant" functionality is retained, but is a no-op for versioned draft stages now due to the existing canView checks
• [x] This functionality works consistently between the core module and any CWP customisations
• [x] Developers can opt in to their own canView handling on a per-classtree basis, with a big warning label about implications and clear documentation that minimises room for error
• [x] There is documentation both for CWP and generic module upgrade paths
  • [x] These docs note that this implementation has edgecases and cannot be relied upon solely
• [ ] Performance is measured against a large dataset on real-world environments (10,000 records, CWP + SC) - Steve: do we still want this? Based on https://github.com/silverstripe/silverstripe-fulltextsearch/pull/276 we can roughly expect solr_reindex to double on basic pages, though basic pages index very fast. What really messes up reindex times is tika text-extraction on pdf's/docx's which will barely be impacted by an extra canView() check.

Removed ACs

• [x] InheritedPermissions is leveraged to reduce performance impact - Steve: found that this would have lead to potential security issues
• [x] Any documentation advising to exclude draft variants is removed - Steve: retaining this, as you may still wish to exclude draft variant if you skip the canView() check

Subtasks

• [x] Introduce canView condition in appropriate places
• [x] Add configuration to allow disabling condition
• [x] Test create, edit, protect, delete operations to ensure the correct behaviour is observed
• [x] Document the change in behaviour, and how to disable it

PRs

• [ ] #281
• [ ] https://github.com/silverstripe/cwp/pull/266

This also has incremental deletion of records; Rather than deleting all records in solr up front, it'll simply delete all records that it KNOWS won't exist up front, and then deletes all other records incrementally in batches.

Each batch is derived using the formula ID % total_groups = this_group. This means that batches don't need to store a list of IDs for each record, and copes with cases where ids may not be contiguous, as well as deleted IDs that may not have a local record anymore.

Re-index is broken into a base implementation, with separate subclasses for each implementation (including immediate reindex for local, message queue and queuedjobs).

Solr_Reindex, two queued jobs, a new logging API, and several tests have been added.

Part of the logic for generating SOLR query syntax has been moved out of SolrIndex and into SearchQuery. This is so that SearchQuery may be used to generate syntax for deletion of records, which is necessary for the incremental deletion feature.

Issue: https://github.com/silverstripe/silverstripe-fulltextsearch/issues/252

Adds a canView() check on everything going in to a solr index and on most implementations will also check it on the way out too

This is PR no.2 for this issue, PR no.1 (merged) added a ShowInSearch check https://github.com/silverstripe/silverstripe-fulltextsearch/pull/275

This PR in a continuation of the work previously done in this PR https://github.com/silverstripe/silverstripe-fulltextsearch/pull/276

singleton($step['class'])->extend('augmentSQL', $sql); in SearchIndex::getDirtyIDs() function breaks some DataExtension objects when calling to their function augmentSQL(), eg. Translatable, or SiteTreeSubsites, since Translatable::augumentSQL(SQLQuery &$query, DataQuery &$dataQuery = null) is expecting the second parameter is a DataQuery object and directly uses it to call its function getQueryParam() in its implementation without checking if $dataQuery is a non-object or not

So this issue could be either fixed in Translatable class or SearchIndex class. Giving that if we fix here in SearchIndex, non of those DataExtension classes need to do anything, I hope it could fixed here.

Scenario I create three different pages. On each of them I have put one of: gdp, jel, knp (unique words so I know these aren't anywhere else on the site) I run a Solr_Reindex I search for "gdp" and just that one page is returned in the results. Searching any of the unique words returns the correct result.

Now, I'd like to make these words synonyms. I go into the Fulltext Search tab in Settings of the CMS (which uses CWP's SynonymsSiteConfig) This config modifies the synonyms.txt file. I enter gdp, jel, knpas my Solr Synonyms. I run a Solr_Configure.

Result Now when I search any of the unique words, I get no results. I do a Reindex just to see if that has an effect. Still no results.

Expected Result What I would expect, is that if I searched for any of those words that all 3 pages would be returned.

Am I misunderstanding the concept of Solr Synonyms? Or do they not function correctly in the module? An issue with version of Solr used in fulltextsearch-local? Or something else?

Just thought I would get the convo started and see if anyone else is having this issue. 😄

Purpose

Currently in fulltextsearch we can filter and we can exclude, but both of these are "single level", in that you cannot group filters with combinations of AND and OR statements.

In order to allow for more complex filtering, I have implemented criterion objects. These objects can be joined and grouped in any number of ways - allowing you to achieve any kind of nested filter that you might want.

New objects

• SearchCriteriaInterface: Interface for SearchCriterion and SearchCriteria classes.
• SearchCriterion: An object containing a single field filter (target field, comparison value, comparison type).
• SearchCriteria: An object containing a collection of SearchCriterion and/or SearchCriteria with conjunctions (IE: AND, OR) between each.
• SearchQueryWriter: A class used to generate a query string based on a SearchCriterion.
• SearchAdapterInterface: An Interface for our Search. This adapter will control what SearchQueryWriter is used for each SearchCriteria.

General usage

We need 3 things to create a SearchCriterion:

• Target: EG the field in our Search Index that we want to filter against.
• Value: The value we want to use for comparison.
• Comparison: The type of comparison (EG: EQUAL, IN, etc).

All currently supported comparisons can be found as constants in SearchCriterion.

Creating a new SearchCriterion

Method 1a and 1b

// `EQUAL` is the default comparison for `SearchCriterion`, so no third param is required.
$criterion = new SearchCriterion('Product_Title', 'My Product');

// Or use the `create` static method.
$criterion = SearchCriterion::create('Product_Title', 'My Product');

Creating a new SearchCriteria

SearchCriteria has a property called $clauses which is a collection of SearchCriterion (above) and/or SearchCriteria (allowing for infinite nesting of clauses), along with the conjunction used between each clause (IE: AND, OR). We want to build up our SearchCriteria by adding to it's $clauses collection.

SearchCriteria can either be passed an object that implements SearchCriteriaInterface, or it can be passed the Target, Value, and Comparison (like above).

Method 1

Instantiate a new SearchCriteria by providing an already instantiated SearchCriterion object. This $criterion will be added as the first item in the $clauses collection.

$criteria = SearchCriteria::create($criterion);

Method 2

Instantiate a new SearchCriteria objects and define the Target, Value, and Comparison. SearchCriteria will create a new SearchCriterion object based on the values, and add it to the $clauses collection.

$criteria = SearchCriteria::create('Product_CatID', array(21, 24, 25), SearchCriterion::IN);

Adding additional SearchCriterion to our SearchCriteria

When you want to add more complexity to your SearchCriteria, there are two methods available:

• addAnd: Add a new SearchCriterion or SearchCriteria with an AND conjunction.
• addOr: Add a new SearchCriterion or SearchCriteria with an OR conjunction.

Method 1

Use method chaining to create a SearchCriterion with two clauses.

// Filter by products with stock that are in either of these 3 categories.
$criteria = SearchCriteria::create('Product_CatID', array(21, 24, 25), SearchCriterion::IN)
    ->addAnd('Product_Stock', 0, SearchCriterion::GREATER_THAN);

Method 2

Systematically add clauses to your already instantiated SearchCriteria.

// Filter by products in either of these 3 categories.
$criteria = SearchCriteria::create('Product_CatID', array(21, 24, 25), SearchCriterion::IN);

... other stuff

// Filter by products with stock.
$criteria->addAnd('Product_StockLevel', 0, SearchCriterion::GREATER_THAN);

Adding multiple levels of filtering to our SearchCriteria

SearchCriteria also allows you to pass in other SearchCriteria objects as you instantiate it and as you use the addAnd and addOr methods.

// Filter by products that are in either of these 3 categories with stock.
$stockCategoryCriteria = SearchCriteria::create('Product_CatID', array(21, 24, 25), SearchCriterion::IN)
    ->addAnd('Product_Stock', 0, SearchCriterion::GREATER_THAN);

// Filter by products in Category ID  1 with stock over 5.
$legoCriteria = SearchCriteria::create('Product_CatID', 1, SearchCriterion::EQUAL)
    ->addAnd('Product_Stock', 5, SearchCriterion::GREATER_THAN);

// Combine the two criteria with an `OR` conjunction
$criteria = SearchCriteria::create($stockCategoryCriteria)
    ->addOr($legoCriteria);

The end result is that you get any products that are in categories 21, 24, 25 that have stock, and any products in category 1 with more than 5 in stock.

Adding SearchCriteria to our SearchQuery

Our SearchQuery class now has a property called $criteria which holds all of our SearchCriteria. You can add new SearchCriteria by using SearchQuery::filterBy().

Method 1

Pass in an already instantiated SearchCriteria object. If you implemented complex filtering (above), you will probably need to follow this method - fully creating your SearchCriteria first, and then passing it to the SearchQuery.

$query->filterBy($criteria);

Method 2a

Where basic (single level) filtering is ok, the SearchQuery::filterBy() method can be used to create your SearchCriterion and SearchCriteria object.

$query->filterBy('Product_CatID', array(21, 24, 25), SearchCriterion::IN);

Method 2b

The filterBy() method will return the current SearchCriteria, this allows you to method chain the addAnd and addOr methods.

// Filter by products with stock that are in either of these 3 categories.
$searchQuery->filterBy('Product_CategoryID', array(21, 24, 25), SearchCriterion::IN)
            ->addAnd('Product_StockLevel', 0, SearchCriterion::GREATER_THAN);

Each item in the $criteria collection are treated with an AND conjunction (matching current filter/exclude functionality).

Search Query Writers

Provided are 3 different SearchQueryWriters for Solr:

• SolrSearchQueryWriter_Basic
• SolrSearchQueryWriter_In
• SolrSearchQueryWriter_Range

When these Writers are provided a SearchCriterion, they will generate the desired query string.

Search Adapters

Search Adapters need to provide the following information:

• What is the search engine's conjunction strings? (EG: are they "AND" and "OR", or are they "&&" and "||", etc).
• What is the desired comparison container string? (EG: "+( query here )") for Solr).
• Most importantly - how to generate the query string from a SearchCriterion.

The SolrSearchAdapter uses SearchQueryWriters (above) to generate query strings from a SearchCriterion.

I realise that having SearchQueryWriters is an additional step, and that usually the Adapter would do the generating (all by itself), but I really wanted to give devs a way to bypass the default "writing" behaviour if they needed to.

Customising your SearchCriterion/SearchQueryWriter

If you find that you do not want your SearchCriterion being parsed by one of the default SearchQueryWriters (for whatever reason), you can optionally pass your own SearchQueryWriter to your SearchCriterion either as the fourth parameter when instantiating it, or by calling setSearchQueryWriter().

If this value is set, then the (default Solr) Adapter will always use the provided SearchQueryWriter, rather than deciding for itself.

This should allow you to have full control over how your query strings are being generated if the default SearchQueryWriters are not cutting it for you.

I'm trying to index a many_many field so that I can filter on it. This used to work in 2.3.5 (or thereabouts) but no longer works in 3.0

public function init() {
   $this->addClass(MyPage::class);
...
   $this->addFilterField('Categories.ID', 'Int');
...
}

...
    private static $many_many = [
      'Categories' => TaxonomyTerm::class,
   ]
...

The schema gets generated correctly: <field name='MyPage_Categories_ID' type='tint' indexed='true' stored='true' multiValued='true'/> but the data never appears on the record in Solr. This means it's impossible to filter on it at query time.

One of my colleagues has experienced a similar problem, and found that one resolution is to rollback to 2.3.5. Unfortunately, my project is on SS4 and I don't have this option.

In the following commit https://github.com/silverstripe/silverstripe-fulltextsearch/commit/34e0594e48f8f5c140e3b41544d39c8f88184114 we allow Symfony 3.2 or 4 however the Process class constructor in these 2 versions works differently.

In Symfony Process 3.2 the constructor accepts a string for the commandline parameter, in Symfony Process 4 the constructor requries an array for the commandline parameter.

SolrReindexImmediateHandler.php sends the command line as a array (previous versions sent it as a string)

Following on from/coming back to https://github.com/silverstripe/silverstripe-fulltextsearch/pull/278 and https://github.com/silverstripe/silverstripe-fulltextsearch/issues/274, it seems either something changed in Silverstripe Cloud Platform or the change in the merged PR worked intermittently/by accident, but there were several sites with issues relating to being able to find newly published content changes through the search on sites on SCP. Full reindex works, but publishing a page doesn't cause the content change to become visible in search.

The latest advice from SCP support is to enable the openSearcher option within the autoCommit config, causing the searchers to reload after each auto commit. The setting can be enabled by copying the solrconfig.xml into the project code, adjusting the config there to override the default, and pointing Solr to it to use when configuring the core as a workaround.

<autoCommit>
    <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
    <openSearcher>true</openSearcher>
</autoCommit>

Should this become the default? Would it have any unwanted implications for systems/platforms, possibly loading up new searchers too many times?

On a large site it is reasonable it will also have a large index (or many small indices) amounting to a large and varied number of fields being indexed on a document.

SearchIndex::__construct calls init which in turn is intended to complete a full build of the index structure. The important components of this are addClass and addFulltextField, the latter of which relies upon fieldData which uses a complex series of introspection calls in nested foreach loops.

The consequence of this is that adding a single field of a not very complicated page type to a search index can take an extreme amount of execution time, e.g. nearly 4 seconds and over 350000 function calls to do it.

The consequence of this can be e.g. HTTP 500.

This is of particular bother as every index is built on every write procedure. E.g. browsing the CMS triggers a write operation for SilverStripe\Security\RememberLoginHash, which in turn is scanned by FulltextSearch to see if it needs to update an index (also in complete pointlessness, I should hope this is never indexed data for the purpose of site search).

Weirdly this seems to affect POST submissions much more than GET, and for some particular pages not others (e.g. UserDefinedForm page). But is a performance issue 100% of the time regardless of the end result in the specific use case.

See cross-post issue on silverstripe/cwp-search: https://github.com/silverstripe/cwp-search/issues/25

This module uses file_get_contents() to post/retrieve data from Solr in some instances. It shouldn't do so, as some servers may have allow_url_fopen disabled in php.ini.

Instead, use of Guzzle (or raw curl) is encouraged for security reasons, mainly to prevent accidental remote code execution/remote file inclusion bugs.

Note that this module explicitly isn't susceptible to RFI vulnerabilities as far as I can tell, but if you're trying to use the module on a hardened server this config value is likely disabled.

edit: Also, renaming the variable from $targetDir would help avoid doubt about whether or not it's a URL. Suggested name: $targetUrl

CWP 2.3.x-dev with demo site sample data

If you search for "CWP" it returns a list of results containing CWP 2.0 info. One example shows that 2.0 is always expanded to 2. 0 in context summaries:

Not matching some characters:

• When searching for CWP 2 you get results, but if you enter CWP 2. you get none, and the same with CWP 2.0
• You can't search for a word with a hyphen in it

I am receiving an error from Solr when trying to filter results on the Content field of SiteTree. Specifically:

org.apache.solr.common.SolrException: undefined field SilverStripeCMSModelSiteTree_Content

I am adding the filter by using the SearchQuery class, like so: (simplified example)

$query = new SearchQuery();
$query->addFilter(SiteTree::class . '_Content', 'filter text');

$index->search($query);

It looks like this issue is caused due to a missing call to SolrIndex->sanitiseClassName in SolrIndex->getRequireFiltersComponent.

You can see an example of where the field name is being sanitised correctly in SolrIndex->getExcludeFiltersComponent here: https://github.com/silverstripe/silverstripe-fulltextsearch/blob/master/src/Solr/SolrIndex.php#L965-L970 which is what lead me to believe it was missing in getRequireFiltersComponent

Edit: I am using silverstripe/fulltextsearch version 3.4.1 but this issue appears to affect newer versions too