How parsing builds the index#

Log Explorer turns a flat log file into structured rows with parsed columns, searchable facets, and pattern-aware groupings. This page explains what happens between the moment you drop a file on the workspace and the moment the events table is ready to query. Use it to predict how the tool behaves on your logs before you load them.

The parsing pipeline#

Every loaded file moves through the same stages. Each stage feeds the next. The workspace becomes interactive only after the full pipeline catches up with your file.

Each arrow is one-way. Log Explorer never changes the source file on disk.

Open and decompress a file#

Drop a file on the workspace, or choose it from the file picker. Log Explorer opens the file in a new tab and names the tab after the file. If the file is compressed, for example a .gz or .zip archive, Log Explorer expands it for you. You do not have to unpack it first. Log Explorer leaves the original file on your disk untouched, and the expanded content stays inside the session.

Log Explorer streams each file as it reads it, so you can search and filter before a very large file finishes loading. The result count in the top right of the query bar updates as more rows become available.

Detect the log format#

After decompression, Log Explorer reads the first lines of the file to decide what it is. It recognises the common structured formats, for example:

• W3C extended: used by many web proxies, where a header line that begins with #Fields: lists the column names and every subsequent line is a space-separated record.
• JSON lines: one self-describing JSON object per line.
• Delimited text: CSV and TSV files with or without a header row.
• Syslog-style plain text: a timestamp followed by a host, a tag, and a free-form message.

If the file holds an explicit field declaration, such as the W3C #Fields: line, those names become the column names of the events table. If it does not, Log Explorer proposes sensible defaults based on the detected shape.

Extract fields from each line#

After it knows the format, Log Explorer parses each line into a row with a value for every known column. Empty fields appear as blanks. The events table, the facets panel on the left, and the by pill on the query bar all show the same set of columns: the field names that the file surfaced during this stage.

Columns keep their types where possible. Numerical fields such as time_taken, sc_status, or cs_bytes sort, range, and aggregate as numbers rather than strings.

Index rows for search#

Log Explorer loads the parsed rows into an in-session index. The index powers the whole workspace at once:

• The events table reads rows from the index in sort order.
• The facets panel reads the distinct values and counts per field.
• The activity histogram reads timestamp buckets.
• The Search for bar translates each field:value expression into an index lookup.
• The Group into modes aggregate over the index.

The index lives in the session and disappears when you close the tab. Log Explorer uploads nothing. Every parsed row stays on your machine.

How LogMine groups similar lines#

Switch Group into to Patterns and select the LogMine sub-tab. Log Explorer then discovers the shapes of the log lines in the file. You do not need to supply a regular expression, because LogMine builds the shapes for you.

LogMine works as follows:

1. It reads each line as a sequence of tokens: words, numbers, punctuation.
2. It compares each line against a growing list of templates. It then decides whether the line matches an existing template closely enough, or whether it seeds a new one.
3. It replaces the tokens that vary from one line to the next (numbers, IPs, identifiers, paths) with a placeholder. It keeps the stable words the same.
4. It produces a compact list of templates. Each template covers some number of the original rows.

The Patterns table then shows one row per template with a count, a volume sparkline, and the template text. Select a small-count template to find the unusual lines, the ones that did not match the common shapes, without a scroll through millions of rows.

When to use LogMine#

• You have a file in an unknown or semi-structured format.
• You want an inventory of the kinds of lines the file contains.
• You hunt for outliers and want the long tail of rare templates.

How Grok matches a known format#

The Grok sub-tab under Patterns works the other way around. It does not discover templates. Instead, you apply templates you already have.

A Grok pattern is a short expression that names the parts of a line you expect to see. For example, the pattern %{IP:client} %{WORD:method} %{URIPATH:path} tells Log Explorer this: the first token is an IP to call client, the next is a word to call method, and the one after that is a URL path to call path. Log Explorer decomposes every matching line into those named fields.

You can chain simpler patterns into more elaborate ones. Log Explorer supplies the standard library of common building blocks (IP addresses, numbers, timestamps, URLs, quoted strings, and so on), so you do not have to write the low-level expressions yourself.

When to use Grok#

• You already know what the lines look like, for example a specific access log format from a service you own.
• You need field names and types out of lines that are not already in a structured format.
• You want the extracted fields to behave like any other column: facetable, groupable, and aggregatable.

How transactions are built#

Switch Group into to Transactions. Log Explorer groups the parsed events by a key of your choice, the value of the by pill, for example cs_username. For each key, Log Explorer computes:

• A count of events with that key.
• A duration, the time between the earliest and latest event in the group.
• A first seen and last seen timestamp.

The rollup runs on the same indexed rows as the rest of the workspace. Any filter in the Search for bar narrows the events first, and only the surviving events feed the transactions table. You can therefore scope a transaction view to any query, for example "transactions by username, for events from this IP, in this hour", without leaving the workspace.

What the pipeline means in practice#

The pipeline above has a few consequences that are worth knowing up front:

• Large files are fine. A streaming parse and an in-session index let you open multi-gigabyte files without pre-processing.
• Compressed files open directly. You do not have to decompress .gz or .zip logs yourself.
• Unknown formats still work. Even when Log Explorer cannot split a line into named columns, LogMine groups similar lines and makes the file explorable.
• Your data stays local. The parse, the index, and every query run on your machine. When you close the tab, Log Explorer discards the session.