solr tutorial - Wikitechy

How to optimize Apache Solr boost factors using genetic algorithms ?

Editor — Mon, 12 Jul 2021 15:37:48 +0000

Optimize Apache Solr boost Factors:

The problem to solve:

When running search indexes with Solr, you might stumble upon is that you have various fields in your documents and you’ve to adjust their weights to get reasonable results.

Finding those “boosting” values can be quite complex when you have many fields and many scenarios.
Usually getting the values right is a task for very experienced integrators.

 
/Solr/select?defType=dismax&q=my+query
&qf=title^**42**+description^**23**+footnotes^**5**+dalmatiners^**101**+foo^**9001**+comments

Looking at it from a more technical perspective – when your Solr query looks like the one above, the question you’ve to answer is how the values for the highlighted numbers should look like to get reasonable results.

Measuring “reasonable”:

- In order to solve the problem answer what we expect the outcome to look like.
- In other words, we’ve to measure how reasonable a specific solution is.
  - For a search engine this can be done with some sample queries and some expectations along with that.

The expectation could come in a form that we explicitly tell which documents we expect in the result lists of specific queries (and at predefined positions).
We can simple test agains the expectations and check whether or not specific boost factor values actually satisfy them.

How to link Apache Solr to MongoDB ?

Editor — Mon, 12 Jul 2021 15:29:04 +0000

Installation

New in mongo-connector 2.5.0, to install mongo-connector with the Solr-doc-manager run:

Pip install 'mongo-connector[solr]'

Setup

Create Solr cores

Make sure the LukeRequestHandler is enabled

This line should be present in your solrconfig.xml file:

Set up your Schema

Mongo Connector stores metadata in every document to help handle rollbacks.
To support these data, you’ll need to include the following to your schema.xml:

The Basics:

Mongo Connector can replicate to Apache Solr search engine by using the Solr DocManager.
To start the connector, you must pass in the base URL for the Solr core to which you want to synchronize.

The most basic usage is the following:

Mongo-connector -m localhost:27017 -t http://localhost:8983/solr -d solr_doc_manager

Mongo Connector and schema.xml

Configuring Solr

Refer to the Apache documentation for configuring Solr and SolrCloud.

N.B.: Key Names and Document Flattening

Mongo Connector automatically “flattens” MongoDB documents.
Fields within sub-documents can be referenced by their “dot-separated path” within the document.
Likewise, array fields are unrolled, so that individual elements are accessible by the field’s original name, plus a “.”, plus the index within the array that the element occupied.

An example:

{
    "subdoc": {
        "a": 1,
        "b": 2,
        "c": 3,
        "array": [
            {"name": "elmo"},
            {"name": "oscar"}
        ]
    }
}

It will become the following in Solr:

{
    "subdoc.a": 1,
    "subdoc.b": 2,
    "subdoc.c": 3,
    "subdoc.array.0.name": "elmo",
    "subdoc.array.1.name": "oscar"
}

Schema.xml

In addition, Mongo Connector comes with an example schema.xml file that can help get you started integrating MongoDB with Apache Solr search.
Solr reads schema.xml in order to find field types, fields that documents may have the primary key, and more.
MongoDB Connector try to acquire the schema for Apache Solr by using the LukeRequestHandler at a particular URI admin/luke/?show=schema&wt=json which is appended to the base of the Apache Solr URL.
So, in the above example, Mongo Connector tries to get the schema for Apache Solr by sending a GET request to http://localhost:8983/solr/admin/luke/?show=schema&wt=json.
Mongo Connector will drop fields from MongoDB documents that are not confirmed in your Apache Solr core’s schema in order to avoid Solr throw exceptions and fail to add those documents.
If you do not describe the fields then you desire in schema.xml and reload the Apache Solr core, Mongo Connector will merrily continue stripping your MongoDB documents of the offending fields.
You can able to check what Apache Solr thinks the schema to your core by visiting the aforementioned endpoint in browser.

What is the most ideal approach to use Apache Solr for relational JOIN ?

Editor — Mon, 12 Jul 2021 09:28:47 +0000

Apache Solr for relational JOIN:

Run a single-use script that do the joins on the SOLR index (file) to replace the author_id and author_name, or include the field if you need both. Updating 10,000,000+ documents will get a while, however it is definitely achievable.
After that you can have new documents indexed with both fields, doing the JOIN one at a time.
In common, SOLR works with entirely denormalized information.
If it is more than just authors’ names, you could still denormalize the information, repeating your documents for every one of the related author documents.
Another technique is SOLR 4 has “JOIN” ability.
You’d first need to at least contain a pipeline wherever author information is indexed into SOLR in real-time.
Utilize Elastic Search to simply deal with the complex SOLR queries involving JOIN-like operations.

What is the internal architecture of Apache solr ?

Editor — Mon, 12 Jul 2021 09:24:15 +0000

Solr Architecture ─ Building Blocks

Following are the major building blocks of Apache Solr:

Solr Architecture ─ Building Blocks Request Handler:

The requests that send to Solr are processed by the request handlers.
The requests may be a query requests otherwise index update requests.
It is based on our requirement; we need to choose the request handler.
To pass a request to Solr, we generally map the handler to a certain URI end-point and the particular request will be served by it.

Search Component:

The search component is a type of search provided in Apache Solr.
It might be spell checking, faceting, query, and hit highlighting and so on.
These search components are registered as search handlers. Multiple components can be registered to a search handler.

Query Parser:

The Apache Solr query parser parses the queries that we pass to Solr and verifies the queries for syntactical errors.
After parsing the queries, it translates them to a format which Lucene understands.

Response Writer:

A response writer is the component which generates the formatted outcome for the user queries.
Apache Solr supports response formats some are XML, JSON, CSV, etc.
We have different response writers for each type of response.

Analyser/tokenizer:

Lucene recognizes data in the form of tokens.
Apache Solr analyses the content, divides it into tokens, and passes these tokens to Lucene.
An analyser examines the text of fields and generates a token stream.
A tokenizer breaks the token stream prepared by the analyser into tokens.

Update Request Processor:

Whenever we send an update request to Apache Solr, the request is run through a set of plugins (signature, logging, indexing), collectively known as update request processor.
This processor is responsible for modifications that are adding a field, dropping a field, etc.