Sign In Start Free Trial

Lucene 4 Cookbook

By : Edwood Ng, Vineeth Mohan

3.2 (5)

Lucene 4 Cookbook

3.2 (5)

By: Edwood Ng, Vineeth Mohan

Overview of this book

This book is for software developers who are new to Lucene and who want to explore the more advanced topics to build a search engine. Knowledge of Java is necessary to follow the code samples. You will learn core concepts, best practices, and also advanced features, in order to build an effective search application.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Sections

Conventions

Reader feedback

Customer support

Free Chapter

1. Introducing Lucene

1. Introducing Lucene

Introduction

Installing Lucene

Setting up a simple Java Lucene project

Obtaining an IndexWriter

Creating an analyzer

Creating fields

Creating and writing documents to an index

Deleting documents

Obtaining an IndexSearcher

Creating queries with the Lucene QueryParser

Performing a search

Enumerating results

2. Analyzing Your Text

2. Analyzing Your Text

Introduction

Obtaining a common analyzer

Obtaining a TokenStream

Obtaining TokenAttribute values

Using PositionIncrementAttribute

Using PerFieldAnalyzerWrapper

Defining custom TokenFilters

Defining custom analyzers

Defining custom tokenizers

Defining custom attributes

3. Indexing Your Data

3. Indexing Your Data

Introduction

Obtaining an IndexWriter

Creating a StringField

Creating a TextField

Creating a numeric field

Creating a DocValue Field

Transactional commits and index versioning

Reusing field and document objects per thread

Delving into field norms

Changing similarity implementation used during indexing

4. Searching Your Indexes

4. Searching Your Indexes

Introduction

Obtaining IndexReaders

Un-inverting single-valued fields in memory with FieldCache

TermVectors

IndexSearcher

Constructing queries

Specifying sort logic

Forming a search result

Pagination

Using Collectors

Sorting with custom FieldComparator

5. Near Real-time Searching

5. Near Real-time Searching

Introduction

Using the DirectoryReader to open index in Near Real-Time

Using the SearcherManager to refresh IndexSearcher

Generational indexing with TrackingIndexWriter

Maintaining search sessions with SearcherLifetimeManager

Performance tuning: latency and throughput

6. Querying and Filtering Data

6. Querying and Filtering Data

Introduction

Performing advanced filtering

Creating a custom filter

Searching with QueryParser

TermQuery and TermRangeQuery

BooleanQuery

PrefixQuery and WildcardQuery

PhraseQuery and MultiPhraseQuery

FuzzyQuery

NumericRangeQuery

DisjunctionMaxQuery

RegexpQuery

SpanQuery

CustomScoreQuery

7. Flexible Scoring

7. Flexible Scoring

Introduction

Overriding similarity

Implementing the BM25 model

Implementing the language model

Implementing the divergence from randomness model

Implementing the information-based model

8. Introducing Elasticsearch

8. Introducing Elasticsearch

Introduction

Getting Elasticsearch

Creating a new index

Predefine field mappings

Adding a document

Deleting a document

Updating a document

Performing bulk indexing

Searching the index

Scaling Elasticsearch

9. Extending Lucene with Modules

9. Extending Lucene with Modules

Introduction

Exploring spatial search

Implementing joins

Performing faceting

Implementing grouping

Employing autosuggest

Implementing highlighting

Index

Index

Customer Reviews

3.2 (5)

5 star

20%

4 star

20%

3 star

40%

2 star

0

1 star

20%

Creating fields

We have learned that indexing information in Lucene requires the creation of document objects. A Lucene document contains one or more field where each one represents a single data point about the document. A field can be a title, description, article ID, and so on. In this section, we will show you the basic structure and how to create a field.

A Lucene field has three attributes:

Name
Type
Value

Name and value are self-explanatory. You can think of a name as a column name in a table, and value as a value in one of the records where record itself is a document. Type determines how the field is treated. You can set FieldType to control whether to store value, to index it or even tokenize text. A Lucene field can hold the following:

String
Reader or preanalyzed TokenStream
Binary(byte[])
Numeric value

How to do it...

This code snippet shows you how to create a simple TextField:

  Document doc = new Document();
  String text = "Lucene is an Information Retrieval library written in Java.";
  doc.add(new TextField("fieldname", text, Field.Store.YES));

How It Works

In this scenario, we create a document object, initialize a text, and add a field by creating a TextField object. We also configure the field to store a value so it can be retrieved during a search.

A Lucene document is a collection of field objects. A field is the name of the value pairs, which you may add to the document. A field is created by simply instantiating one of the Field classes. Field can be inserted into a document via the add method.

Search

Your notes and bookmarks