# `Nebulex.Adapters.Multilevel`
[🔗](https://github.com/elixir-nebulex/nebulex_distributed/blob/v3.2.3/lib/nebulex/adapters/multilevel.ex#L1)

Adapter module for the multi-level cache topology.

The Multi-level adapter is a simple layer that works on top of a local or
distributed cache implementation, enabling a cache hierarchy by levels.
Multi-level caches generally operate by checking the fastest,
level 1 (L1) cache first; if it hits, the adapter proceeds at
high speed. If that first cache misses, the next fastest cache
(level 2, L2) is checked, and so on, before accessing external
memory (that can be handled by a `cacheable` decorator).

For write functions, the "Write Through" policy is applied by default;
this policy ensures that the data is stored safely as it is written
throughout the hierarchy. However, it is possible to force the write
operation in a specific level (although it is not recommended) via
`:level` option, where the value is a positive integer greater than 0.

## How Multi-Level Caches Work

A multi-level cache is organized in a hierarchy of cache layers, where each
layer is typically faster but smaller than the next. The most common pattern
is a **near-cache topology** with two levels:

  * **L1 (Level 1)**: Local cache - Fast, in-process, limited size. Examples:
    `Nebulex.Adapters.Local`.
  * **L2 (Level 2)**: Distributed/remote cache - Slower, shared across nodes,
    larger capacity. Examples: `Nebulex.Adapters.Partitioned`, Redis, etc.

Multi-level caches provide **performance improvement** by reducing latency
for frequently accessed data (served from L1) while maintaining large capacity
through the L2 layer.

```asciidoc
                   ┌──────────────┐
                   │    Client    │
                   └──────┬───────┘
                          │
                 ┌────────┴─────────┐
                 │    L1 (Local)    │◄── Fast, limited size
                 └────────┬─────────┘
                          │
                 ┌────────┴─────────┐
                 │ L2 (Distributed) │◄── Slower, larger capacity
                 └──────────────────┘

Read:  L1 → L2 (replicate back on hit, inclusive mode)
Write: L1 → L2 (write-through to all levels)
```

### Cache Lookup (Read Operations)

When you perform a **read operation** (e.g., `get`, `fetch`), the adapter
checks the cache levels in order (L1 → L2 → L3, etc.) and returns the value
from the **first level that contains it**:

  1. Check L1 (fast, local cache).
  2. If found → Return immediately.
  3. If not found → Check L2.
  4. If found in L2 → Return and replicate to L1 (if inclusive mode).
  5. If not found → Check L3, L4, etc.
  6. If not found in any level → Return error or miss.

This **single-hop lookup** pattern ensures you always get data from the
fastest available level while maintaining a fallback to slower (but larger)
levels.

**Example:**
- Request for key "user:123"
- L1 miss (not in local cache)
- L2 hit (found in distributed cache)
- Value returned to client
- In inclusive mode: value is replicated back to L1 for future requests

### Write-Through Policy

The multi-level adapter uses a **Write-Through** policy for write operations.
This means data is written to **all levels synchronously** before returning
to the caller:

  1. Write to L1 (local cache).
  2. Write to L2 (distributed cache).
  3. Write to L3, L4, etc. (if any)
  4. Return to caller.

This ensures:

  * **Consistency**: Data is immediately available in all levels.
  * **Safety**: Data survives L1 failures (protected in L2).
  * **Simplicity**: No complex cache invalidation logic needed.

**Trade-off**: Write operations are **slower** because they must complete on
all levels before returning. However, this is typically acceptable when:
  - Read operations are usually more frequent.
  - Distributed writes are less latency-sensitive than reads.
  - Consistency guarantees are worth the cost.

**Failure Handling**: If a write fails at any level, the operation stops
(fail-fast). Partially written data at earlier levels is **not rolled back**
(atomic writes are not guaranteed across levels).

### Replication in Inclusive Mode

When using the **`:inclusive` policy** (default), data can exist in multiple
levels simultaneously. The adapter automatically **replicates data backward**
(from slower to faster levels) during reads:

  * **Inclusive mode**: Same key can exist in L1, L2, and L3 simultaneously.
  * **Exclusive mode**: Same key can exist in only one level at a time.

#### Inclusive Mode Workflow

  1. **Read (inclusive mode)**:
     - Check L1, L2, L3 in order.
     - If found in L2 (not in L1) → Replicate to L1.
     - Future requests hit L1 (faster).

  2. **Write (inclusive mode)**:
     - Write to L1, L2, L3 (all levels).
     - Future reads hit L1 (fastest).

  3. **Eviction (inclusive mode)**:
     - L1 evicts a key (e.g., due to size limit).
     - Key still exists in L2.
     - Next read finds it in L2 and replicates it back to L1.

#### Exclusive Mode Workflow

  1. **Read (exclusive mode)**:
     - Find key in L2.
     - Return value (do NOT replicate to L1).

  2. **Write (exclusive mode)**:
     - Write to L1, L2, L3 (write-through still applies).
     - Data exists in all levels.

  3. **Eviction (exclusive mode)**:
     - L1 evicts a key.
     - L2 still has it.
     - Next read must go to L2 (no replication).

#### Choosing the Right Mode

  * **Inclusive (default)**: Good for hot-spot data patterns.
    - Pro: Faster subsequent reads (hot data stays in L1).
    - Con: More memory usage (duplicated in multiple levels).
    - Con: `get_all` is slower (requires per-entry replication).

  * **Exclusive**: Good for large data sets or strict memory limits.
    - Pro: Less memory usage (data exists once).
    - Con: Slower reads after eviction (must fetch from L2).
    - Con: More complex consistency management.

### TTL (Time-To-Live) Handling

TTL values are **per-level and independent**:

  * Each level has its own TTL for the same key.
  * When writing with TTL, the same TTL is applied to all levels.
  * Each level independently expires the entry.
  * In inclusive mode: If L1 expires but L2 still has it, next read
    replicates from L2 back to L1 with remaining TTL.

#### Why TTL is Per-Level

Each cache level uses a **different adapter with its own eviction policy**.
For example:

  * **L1 (`Nebulex.Adapters.Local`)**: TTL is evaluated **on-demand** during
    reads and via a **garbage collector** that periodically removes expired
    entries and creates new generations. Behavior is controlled by
    `gc_interval` option.

  * **L2 (`Nebulex.Adapters.Partitioned` or Redis)**: TTL is handled by the
    underlying distributed cache. Redis, for example, uses its own TTL
    expiration mechanism which may differ from the local adapter's approach.

Because each adapter manages TTL differently, the **expiration timing and
behavior can vary significantly across levels**:

  - L1 may expire entries within seconds (controlled by GC interval).
  - L2 may expire entries at slightly different times (depends on Redis TTL).
  - A key can exist in L2 but be expired in L1.
  - The remaining TTL is automatically synchronized when data is replicated
    during reads.

This independence is **intentional and necessary** because:
  - Different adapters have different TTL mechanisms.
  - Forcing synchronized TTL across levels would require complex coordination.
  - Allowing independent TTL gives each adapter control over its eviction.
  - The cost is acceptable because reads automatically sync TTL through
    replication.

This means:
  - Data can be available in L2 even after L1 expires it.
  - Evictions are independent per level and controlled by each adapter.
  - TTL synchronization happens automatically during reads
    (in inclusive mode).

We can define a multi-level cache as follows:

    defmodule MyApp.Multilevel do
      use Nebulex.Cache,
        otp_app: :nebulex,
        adapter: Nebulex.Adapters.Multilevel

      defmodule L1 do
        use Nebulex.Cache,
          otp_app: :nebulex,
          adapter: Nebulex.Adapters.Local
      end

      defmodule L2 do
        use Nebulex.Cache,
          otp_app: :nebulex,
          adapter: Nebulex.Adapters.Partitioned
      end
    end

Where the configuration for the cache and its levels must be in your
application environment, usually defined in your `config/config.exs`:

    config :my_app, MyApp.Multilevel,
      inclusion_policy: :inclusive,
      levels: [
        {
          MyApp.Multilevel.L1,
          gc_interval: :timer.hours(12)
        },
        {
          MyApp.Multilevel.L2,
          primary: [
            gc_interval: :timer.hours(12)
          ]
        }
      ]

If your application was generated with a supervisor (by passing `--sup`
to `mix new`) you will have a `lib/my_app/application.ex` file containing
the application start callback that defines and starts your supervisor.
You just need to edit the `start/2` function to start the cache as a
supervisor on your application's supervisor:

    def start(_type, _args) do
      children = [
        {MyApp.Multilevel, []},
        ...
      ]

See `Nebulex.Cache` for more information.

## Options

This adapter supports the following options and all of them can be given via
the cache configuration:

* `:stats` (`t:boolean/0`) - Enables or disables cache statistics collection (default: enabled).

  When enabled, collects hit/miss/write statistics available via the
  `info()` command. Statistics are collected per-level and aggregated at
  the multi-level cache level. See the ["Info API"](#module-info-api)
  section for details on how to access cache statistics.

  The default value is `true`.

* `:levels` (non-empty `t:keyword/0`) - Required. Defines the cache hierarchy as a non-empty keyword list of cache levels.

  Each element must be a tuple of `{cache_module, opts}` where:

    * `cache_module` - The cache module to use for this level
      (e.g., `MyApp.Multilevel.L1`, `MyApp.Multilevel.L2`).
    * `opts` - Keyword list of options passed to that cache's
      `start_link/1`.

  **Level Ordering**: The order of elements determines the hierarchy:
    - First element = L1 (fastest, checked first)
    - Second element = L2 (slower, larger capacity)
    - Nth element = LN (slowest, largest capacity)

  **Example:**

      levels: [
        {MyApp.Multilevel.L1, gc_interval: :timer.hours(12)},
        {MyApp.Multilevel.L2, primary: [gc_interval: :timer.hours(12)]}
      ]

  This option is **required**. If not set or empty, the adapter raises an
  exception. Each level must be a different cache module instance.

* `:inclusion_policy` - Specifies whether the same data can exist in multiple cache levels
  simultaneously (default: inclusive).

  `:inclusive` - Same key can exist in L1, L2, L3, etc. simultaneously.
  On read, if found in L2 but not L1, automatically replicate back to L1
  for faster future reads. Trade-off: Uses more memory (data duplicated
  in multiple levels). The `get_all` operation is slower because each entry
  requires per-entry replication. Use the `:replicate` option to skip
  replication if needed.

  `:exclusive` - Same key can exist in only one level at a time. On read,
  return value WITHOUT replicating to L1. Trade-off: Reads after L1
  eviction must fetch from slower levels. Good for large datasets or
  strict memory constraints.

  See the ["How Multi-Level Caches Work"](#module-how-multi-level-caches-work)
  section for detailed examples and workflow diagrams.

  The default value is `:inclusive`.

## Shared options

Almost all of the cache functions outlined in `Nebulex.Cache` module
accept the following options:

* `:timeout` (`t:timeout/0`) - The time in milliseconds to wait for a command to complete. Set to
  `:infinity` to wait indefinitely.

  **Note**: The timeout applies to each level independently.

  The default value is `5000`.

* `:level` (`t:pos_integer/0`) - An integer greater than 0 that specifies the cache level to execute the
  operation on.

  > #### WARNING {: .warning}
  >
  > Using this option **breaks the multi-level cache semantics**
  > and is **not recommended** for normal operations. It's primarily useful
  > for:
  > - Debugging and testing.
  > - Administrative tasks.
  > - Advanced use cases where you need direct level access.

### Queryable API options

The following options apply to `get_all`, `count_all`, `delete_all`,
and `stream` commands:

* `:replicate` (`t:boolean/0`) - Controls whether entries are replicated backward during `get_all`
  operations (default: replicate).

  Applies only to `get_all` when using `:inclusive` inclusion policy.

  When enabled, entries found in L2 are automatically replicated back to L1
  for faster future reads. Trade-off: Each entry requires a replication
  operation. When disabled, entries are returned without replicating to L1,
  which is faster for bulk reads where replication is unnecessary.

  Example - Fast bulk read without L1 replication:

      MyCache.get_all(:user_ids, replicate: false)

  This only affects `get_all`. Regular `get` always respects the inclusion
  policy. Ignored when using `:exclusive` policy (no replication occurs).

  The default value is `true`.

* `:on_error` (`:raise` | `:nothing`) - Controls error handling during queryable operations (`get_all`,
  `count_all`, `delete_all`, `stream`).

  `:raise` - Raise an exception if any error occurs on any level. Fail-fast
  with no partial results. Use for correctness-critical operations where you
  need guarantee of success or explicit failure.

  `:nothing` - Skip errors silently and continue processing. Returns partial
  results from levels that succeeded. Use for large bulk reads, analytics,
  or best-effort operations where partial results are acceptable.

  Errors can occur from network issues (RPC timeout), level failures,
  unavailability, or data corruption.

  The default value is `:raise`.

## Telemetry events

The multi-level adapter emits Telemetry events for itself and for each
configured cache level. By default, each level gets a unique telemetry
prefix derived from its module name, making events naturally distinguishable.

### Default Behavior

Each cache level is a separate module (e.g., `MyApp.Multilevel.L1`,
`MyApp.Multilevel.L2`), so each gets a unique telemetry prefix by default:

    defmodule MyApp.Multilevel do
      use Nebulex.Cache,
        otp_app: :my_app,
        adapter: Nebulex.Adapters.Multilevel

      defmodule L1 do
        use Nebulex.Cache,
          otp_app: :my_app,
          adapter: Nebulex.Adapters.Local
      end

      defmodule L2 do
        use Nebulex.Cache,
          otp_app: :my_app,
          adapter: Nebulex.Adapters.Partitioned
      end
    end

With this setup, events are naturally separated by module:

  * Multilevel adapter: `[:my_app, :multilevel, :command, :start]`
  * L1 cache (Local): `[:my_app, :multilevel, :l1, :command, :start]`
  * L2 cache (Partitioned): `[:my_app, :multilevel, :l2, :command, :start]`
  * L2 primary storage: `[:my_app, :multilevel, :l2, :primary, :command, :start]`

This default behavior is already good for distinguishing events based on
which cache level emitted them.

### Custom Telemetry Prefixes (Optional)

If you want to override the default prefix for a level, use the optional
`:telemetry_prefix` option:

    config :my_app, MyApp.Multilevel,
      levels: [
        {MyApp.Multilevel.L1, telemetry_prefix: [:my_app, :cache, :l1]},
        {MyApp.Multilevel.L2, telemetry_prefix: [:my_app, :cache, :l2]}
      ]

This is useful if you want:
  * A different naming convention for your telemetry events.
  * To aggregate events from multiple caches under a common prefix.
  * To match an existing telemetry structure in your application.

### Events for Distributed L2 Adapters

If L2 uses a distributed adapter like `Nebulex.Adapters.Partitioned`, you
also get events from its primary storage:

    MyApp.Multilevel.get(key)
    # Emits (with default prefixes):
    #   [:my_app, :multilevel, :command, :start]        # Multilevel wrapper
    #   [:my_app, :multilevel, :l1, :command, :start]   # L1
    #   [:my_app, :multilevel, :l2, :command, :start]   # L2 wrapper
    #   [:my_app, :multilevel, :l2, :primary, :command, :start]  # L2 primary

Refer to the [Telemetry guide](https://hexdocs.pm/nebulex/telemetry.html)
for complete information on Nebulex Telemetry events and how to attach
handlers.

## Info API

As explained above, the multi-level adapter uses the configured cache levels.
Therefore, the information provided by the `info` command will depend on the
adapters configured for each level. The Nebulex built-in adapters support the
recommended keys `:server`, `:memory`, and `:stats`. Additionally, the
multi-level adapter supports:

  * `:levels_info` - A list with the info map for each cache level.

For example, the info for `MyApp.Multilevel` may look like this:

    iex> MyApp.Multilevel.info!()
    %{
      memory: %{total: nil, used: 206760},
      server: %{
        cache_module: MyApp.Multilevel,
        cache_name: :multilevel_inclusive,
        cache_adapter: Nebulex.Adapters.Multilevel,
        cache_pid: #PID<0.998.0>,
        nbx_version: "3.0.0"
      },
      stats: %{
        hits: 0,
        misses: 0,
        writes: 0,
        evictions: 0,
        expirations: 0,
        deletions: 0,
        updates: 0
      },
      levels_info: [
        %{
          memory: %{total: nil, used: 68920},
          server: %{
            cache_module: MyApp.Multilevel.L1,
            cache_name: MyApp.Multilevel.L1,
            cache_adapter: Nebulex.Adapters.Local,
            cache_pid: #PID<0.1000.0>,
            nbx_version: "3.0.0"
          },
          stats: %{
            hits: 0,
            misses: 0,
            writes: 0,
            evictions: 0,
            expirations: 0,
            deletions: 0,
            updates: 0
          }
        },
        %{
          memory: %{total: nil, used: 68920},
          nodes: [:"node1@127.0.0.1"],
          server: %{
            cache_module: MyApp.Multilevel.L2,
            cache_name: MyApp.Multilevel.L2,
            cache_adapter: Nebulex.Adapters.Partitioned,
            cache_pid: #PID<0.1015.0>,
            nbx_version: "3.0.0"
          },
          stats: %{
            hits: 0,
            misses: 0,
            writes: 0,
            evictions: 0,
            expirations: 0,
            deletions: 0,
            updates: 0
          },
          nodes_info: %{
            "node1@127.0.0.1": %{
              memory: %{total: nil, used: 68920},
              server: %{
                cache_module: MyApp.Multilevel.L2.Primary,
                cache_name: MyApp.Multilevel.L2.Primary,
                cache_adapter: Nebulex.Adapters.Local,
                cache_pid: #PID<0.1017.0>,
                nbx_version: "3.0.0"
              },
              stats: %{
                hits: 0,
                misses: 0,
                writes: 0,
                evictions: 0,
                expirations: 0,
                deletions: 0,
                updates: 0
              }
            }
          }
        }
      ]
    }

## Extended API

This adapter provides some additional convenience functions to the
`Nebulex.Cache` API.

### `inclusion_policy/0,1`

Returns the inclusion policy of the cache.

    iex> MyCache.inclusion_policy()
    :inclusive

## Near cache topology example

The multi-level adapter can be used to implement a **near-cache topology**
with different types of cache backends. The most common pattern is L1 (local,
fast) + L2 (distributed, larger capacity).

### L1 (Local) + L2 (Redis or External Cache)

Instead of using `Nebulex.Adapters.Partitioned` for L2, you can use any
external cache system via its adapter. For example, with Redis:

    defmodule MyApp.NearCache do
      use Nebulex.Cache,
        otp_app: :my_app,
        adapter: Nebulex.Adapters.Multilevel

      defmodule L1 do
        use Nebulex.Cache,
          otp_app: :my_app,
          adapter: Nebulex.Adapters.Local
      end

      defmodule L2 do
        use Nebulex.Cache,
          otp_app: :my_app,
          adapter: Nebulex.Adapters.Redis
      end
    end

Configuration:

    config :my_app, MyApp.NearCache,
      levels: [
        {MyApp.NearCache.L1, gc_interval: :timer.hours(1), max_size: 10_000},
        {MyApp.NearCache.L2, conn_opts: [host: "localhost", port: 6379]}
      ]

This topology provides:

  * **L1 (Local)**: In-process cache with fast micro-second latency.
  * **L2 (Redis)**: Shared across nodes, larger capacity, multi-millisecond
    latency.

**Benefits:**
  - Hot data is served from L1 (very fast).
  - Cold data fetches from Redis L2, then cached in L1 on next access.
  - Data survives node restarts (in Redis).
  - Works across multiple nodes with a shared Redis instance.

**Use case**: Web applications where you want blazing-fast L1 performance for
frequently accessed data while using Redis for distributed, shared storage.

## Transactions

The multilevel adapter supports distributed transactions via
`Nebulex.Distributed.Transaction`, which uses Erlang's `:global` module for
cluster-wide lock coordination.

**Automatic Node Detection**: When using transactions with multilevel caches,
the adapter automatically detects distributed levels (e.g., partitioned or
replicated caches) and ensures locks are acquired across all relevant cluster
nodes. This means you don't need to manually specify nodes when using
distributed levels.

**Example**:

    # Multilevel cache with local L1 and partitioned L2
    MyApp.Multilevel.transaction(fn ->
      # Locks are automatically set across all partitioned cache nodes
      counter = MyApp.Multilevel.get!(:counter, default: 0)
      MyApp.Multilevel.put!(:counter, counter + 1)
    end, keys: [:counter])

**Recommendation**: Always specify the `:keys` option to enable fine-grained
locking and maximize concurrency. Without keys, a global lock is used which
serializes all transactions.

See `Nebulex.Distributed.Transaction` for more information about transaction
options, behavior, and performance considerations.

## CAVEATS

Because this adapter reuses other existing/configured adapters, it inherits
all their limitations too. Therefore, it is highly recommended to check the
documentation of the adapters to use.

---

*Consult [api-reference.md](api-reference.md) for complete listing*