Background Jobs

Background jobs improve memory quality over time. They run separately from write() and retrieve() - you schedule them yourself (every few hours, via cron, APScheduler, or a simple loop).

Do I need them?

For getting started: no. The write() and retrieve() pipelines work without background jobs. Your agent will still extract facts, resolve entities, and return relevant context.

For production: yes. Without them, importance scores stay flat (0.5 for everything), entity summaries are never generated, patterns and contradictions go undetected, and retrieval quality degrades over time as the memory grows.

flowchart LR
    A["Scheduler\n(periodic)"] --> B["Clustering"]
    A --> C["Consolidation"]
    A --> D["Importance\nScoring"]
    A --> E["Summary\nRefresh"]
    A --> F["Profile\nConsolidation"]

Overview

arandu provides four categories of background jobs:

Job	Purpose	Uses LLM?	Frequency
Clustering	Group related facts semantically	Yes (summaries)	Every 4-8 hours
Consolidation	Detect patterns, contradictions, trends	Yes	Every 4-8 hours
Entity Profile Consolidation	Rebuild comprehensive entity profiles from all facts	Yes	Every 4-8 hours
Memify	Convert episodic facts to procedural/semantic knowledge	Yes	Daily
Sleep-time compute	Score importance, refresh summaries, detect communities	Partially	Every 4-8 hours

All jobs are exposed as async functions you can call directly or schedule with your preferred task runner (APScheduler, Celery, cron, etc.).

Neuroscience parallel

Background jobs mirror sleep-time processing in the brain. During sleep, the brain consolidates memories, transfers information from hippocampus (short-term) to neocortex (long-term), prunes irrelevant connections, and strengthens important ones. These jobs perform the same operations on your agent's memory.

---

Clustering

In plain English: Groups related facts together. Facts about someone's job, colleagues, and projects end up in one cluster. This makes retrieval more contextual - when you ask about someone's work, the system knows which facts are related.

Fact Clustering

from arandu import cluster_user_facts, ClusteringResult

result: ClusteringResult = await cluster_user_facts(
    session=db_session,
    agent_id="user_123",
    embedding_provider=embedding_provider,
    llm_provider=llm_provider,
    config=memory_config,
)

Legacy function names

Some background job functions retain "user" in their names (e.g., cluster_user_facts, get_entities_for_user). These accept agent_id as their parameter - the names are historical and will be aliased in a future version.

How it works:

1. Groups facts by (entity_type, entity_key) - facts about the same entity stay together
2. Generates a 2-3 sentence summary per cluster using an LLM
3. Computes and stores cluster embeddings for later community detection
4. Idempotent - updates existing clusters rather than creating duplicates

Community Detection

from arandu import detect_communities, CommunityDetectionResult

result: CommunityDetectionResult = await detect_communities(
    session=db_session,
    agent_id="user_123",
    embedding_provider=embedding_provider,
    llm_provider=llm_provider,
    config=memory_config,
)

How it works:

1. Compares cluster embeddings using cosine similarity
2. Groups clusters above community_similarity_threshold (default 0.75)
3. Creates MemoryMetaObservation records with type "community_theme"
4. Example: a "work" community might include clusters about colleagues, projects, and company facts

Configuration

Parameter	Default	Description
cluster_max_age_days	90	Maximum age of facts to include in clustering
community_similarity_threshold	0.75	Cosine similarity threshold for grouping clusters

Walkthrough: before and after clustering

Before clustering - 12 facts about a user, ungrouped:

person:fernanda → "Fernanda works at Orion Tech"
person:fernanda → "Fernanda graduated from Unicamp"
person:fernanda → "Fernanda built the data pipeline"
person:bruno → "Bruno works at Orion Tech"
person:bruno → "Bruno developed the ML model"
person:bruno → "Bruno runs marathons"
person:marcos → "Marcos lives in Porto Alegre"
person:marcos → "Marcos is a product manager"
...

After clustering:

Cluster 1: "Orion Tech engineering team"
  → Fernanda works at Orion Tech
  → Bruno works at Orion Tech
  → Fernanda built the data pipeline
  → Bruno developed the ML model
  Summary: "The Orion Tech data/ML team includes Fernanda (pipeline) and Bruno (ML model)"

Cluster 2: "Marcos's personal life"
  → Marcos lives in Porto Alegre
  → Marcos is married to Carolina
  Summary: "Marcos lives in Porto Alegre with his wife Carolina"

Impact on retrieval: When someone asks "Tell me about the Orion Tech team", the cluster summary and its facts score higher because they're grouped together - the system understands they're related.

---

Consolidation

In plain English: Looks across all recent facts and events to find bigger patterns: "This person mentions running every Monday" (pattern), "They said they live in SP but also in RJ" (contradiction), "Their mood has been improving lately" (trend). Stores these as meta-observations that enrich retrieval.

Periodic Consolidation (L2)

from arandu import run_consolidation, ConsolidationResult

result: ConsolidationResult = await run_consolidation(
    session=db_session,
    agent_id="user_123",
    llm_provider=llm_provider,
    config=memory_config,
)

How it works:

1. Analyzes events and facts over a lookback window (consolidation_lookback_days)
2. Detects patterns across facts:
3. Insights - Emergent understanding from multiple facts
4. Patterns - Repeated behaviors or preferences
5. Contradictions - Conflicting facts that need resolution
6. Trends - Changes over time
7. Generates MemoryMetaObservation records
8. Tags events with emotions (emotion, intensity, energy level)

Profile Consolidation (L3)

from arandu import run_profile_consolidation

await run_profile_consolidation(
    session=db_session,
    agent_id="user_123",
    llm_provider=llm_provider,
)

How it works:

1. Refreshes entity summaries via LLM - a higher-level view of each entity
2. Updates the overall profile overview
3. Triggered periodically (less frequently than L2)

Configuration

Parameter	Default	Description
consolidation_min_events	3	Minimum events before running consolidation
consolidation_lookback_days	7	How far back to look for patterns

Neuroscience parallel

Consolidation mirrors the brain's memory consolidation during sleep. The hippocampus replays recent experiences, the neocortex detects patterns and integrates them into existing knowledge structures, and contradictions are flagged for resolution. L2 consolidation is analogous to slow-wave sleep (SWS) replay, while L3 profile consolidation is analogous to REM sleep's role in integrating memories into semantic knowledge.

---

Entity Profile Consolidation

In plain English: The write pipeline seeds a thin profile when an entity is first seen, but it never overwrites an existing profile. This background job is the authoritative source for comprehensive, up-to-date entity profiles. It reads ALL active facts for each entity and generates a thorough profile covering every major aspect: identity, relationships, interests, activities, events, and preferences.

Run Profile Consolidation

from arandu import consolidate_entity_profiles, ProfileConsolidationResult

result: ProfileConsolidationResult = await consolidate_entity_profiles(
    session=db_session,
    agent_id="user_123",
    llm_provider=llm_provider,
    config=memory_config,
)

print(result.profiles_consolidated)  # Number of profiles rebuilt
print(result.profiles_skipped)       # Entities skipped (no facts or LLM error)

How it works:

1. Finds entities ordered by fact count (highest first), limited to 20 per run
2. Filters to entities whose profile is NULL or older than summary_refresh_interval_days
3. For each entity, loads up to 50 active facts (ordered by importance)
4. Sends all facts to the LLM with instructions to generate a comprehensive profile (100-300 tokens) covering ALL major aspects
5. Saves the profile to memory_entities.profile_text and updates profile_refreshed_at
6. Profiles are always generated in English, regardless of the language of the underlying facts

Write pipeline vs background job:

	Write Pipeline	consolidate_entity_profiles()
When	During write() (synchronous)	Scheduled background job
Scope	Seeds thin profiles for NEW entities only	Rebuilds comprehensive profiles from ALL facts
Overwrites	Never overwrites existing profiles	Refreshes stale profiles
Authority	Initial seed only	Authoritative source for profile_text

Entity profiles are used as context during informed extraction (write pipeline) to help the LLM understand what is already known about an entity. They are NOT injected into retrieval output.

Configuration

Parameter	Default	Description
summary_refresh_interval_days	7	Days before a profile is considered stale and eligible for refresh

---

Memify

In plain English: Over time, specific details ("went to a Python meetup on March 5") become general knowledge ("regularly attends tech meetups"). Memify distills episodic facts into higher-level knowledge and prunes stale facts that haven't been mentioned in a while.

Run Memify

from arandu import run_memify, MemifyResult

result: MemifyResult = await run_memify(
    session=db_session,
    agent_id="user_123",
    llm_provider=llm_provider,
    embedding_provider=embedding_provider,
    config=memory_config,
)

How it works:

1. Groups related facts by entity and topic
2. Generates distilled summaries (procedural/semantic knowledge)
3. Checks vitality - facts mentioned recently are kept; stale facts may be deprecated
4. Merges similar procedures to prevent knowledge fragmentation

Vitality Scoring

from arandu import compute_vitality

# Synchronous, per-fact function — NOT async
score = compute_vitality(fact)          # uses datetime.now(UTC)
score = compute_vitality(fact, now=ts)  # custom timestamp

Vitality measures how "alive" a fact is based on:

• Retrieval (0.30) - Log-scale of times_retrieved
• Recency (0.25) - Exponential decay from last_retrieved_at (30-day half-life)
• Confidence (0.20) - Raw confidence value
• Reinforcement (0.15) - Bounded reinforcement count (up to 5)

Correction penalty: 0.8 ^ user_correction_count. Superseded facts (valid_to is not None) return 0.0.

Neuroscience parallel

Memify mirrors the forgetting curve described by Hermann Ebbinghaus (1885). Memories decay exponentially over time unless reinforced through retrieval practice. Facts with high vitality (frequently accessed) resist decay, while low-vitality facts gradually fade - just as the brain prunes synaptic connections for unused information.

---

Sleep-Time Compute

In plain English: Three maintenance jobs that keep retrieval sharp: (1) score which entities matter most, (2) refresh entity summaries for the important ones, (3) detect communities of related entities. The first one is pure SQL (cheap), the other two use LLM calls.

Job 1: Entity Importance Scoring

from arandu import compute_entity_importance, EntityImportanceResult

result: EntityImportanceResult = await compute_entity_importance(
    session=db_session,
    agent_id="user_123",
    config=memory_config,
)

Pure SQL computation (no LLM calls). Scores each entity from 0.0 to 1.0 using four normalized signals:

Signal	Weight	Description
Fact density	0.30	Number of facts linked to the entity (via MemoryFactEntityLink). Includes facts where the entity is primary subject AND facts that merely mention it.
Recency	0.25	Exponential decay (30-day half-life)
Retrieval frequency	0.25	How often facts about this entity are retrieved
Relationship degree	0.20	Number of incoming + outgoing relationships

The importance score is used as a signal in retrieval scoring and as a priority factor for summary refresh.

Walkthrough: how importance scoring changes retrieval

Before importance scoring - all entities have default importance = 0.5:

person:fernanda  → importance: 0.50 (default)
person:marcos    → importance: 0.50 (default)
organization:vertix → importance: 0.50 (default)
product:xgboost  → importance: 0.50 (default)

After importance scoring:

person:fernanda  → importance: 0.85 (12 facts, recent, high retrieval freq)
organization:vertix → importance: 0.78 (8 facts, many relationships)
person:marcos    → importance: 0.55 (4 facts, moderate activity)
product:xgboost  → importance: 0.25 (1 fact, mentioned once, no relationships)

Impact on retrieval: When two facts have similar semantic scores, the one about Fernanda (importance=0.85) ranks above the one about XGBoost (importance=0.25). This reflects the reality that Fernanda is a central entity in this user's memory while XGBoost is a peripheral detail.

Impact on summary refresh: Fernanda and Vertix get their summaries refreshed first (higher priority). XGBoost may never get a summary - it's too low priority.

Job 2: Entity Summary Refresh

from arandu import refresh_entity_summaries, SummaryRefreshResult

result: SummaryRefreshResult = await refresh_entity_summaries(
    session=db_session,
    agent_id="user_123",
    llm_provider=llm_provider,
    config=memory_config,
)

Refreshes stale entity summaries:

• Stale condition: summary_text IS NULL or last refresh > 7 days ago
• Priority: entities with higher importance_score refreshed first
• Limit: 10 entities per run (prevents timeout)
• Generates 2-3 sentence summaries from the entity's facts using an LLM

Job 3: Entity Community Detection

from arandu import detect_entity_communities

result = await detect_entity_communities(
    session=db_session,
    agent_id="user_123",
    config=memory_config,
)

Connected-component clustering on the entity relationship graph (no LLM calls):

1. Loads active entities and edges (strength >= 0.3)
2. Runs Union-Find (with path compression + union by rank) to find connected components
3. Groups entities into communities (components with >= 2 members)
4. Returns {"communities_found": int, "total_entities_assigned": int}

Neuroscience parallel

Sleep-time compute mirrors offline processing during sleep. The brain doesn't just passively store memories during sleep - it actively reorganizes them. Importance scoring is analogous to the brain's process of synaptic homeostasis (Tononi & Cirelli), where strongly activated synapses are maintained while weakly activated ones are pruned. Summary refresh mirrors the formation of gist memories - compressed representations that capture the essence of detailed episodes.

---

Scheduling

arandu doesn't include a scheduler - you bring your own. All background functions are simple async callables that can be integrated with any scheduling system.

Example: APScheduler

from apscheduler.schedulers.asyncio import AsyncIOScheduler
from arandu import (
    cluster_user_facts,
    consolidate_entity_profiles,
    run_consolidation,
    compute_entity_importance,
    refresh_entity_summaries,
)

scheduler = AsyncIOScheduler()

async def maintenance_cycle():
    async with get_session() as session:
        for agent_id in await get_active_users(session):
            await compute_entity_importance(session, agent_id, config=config)
            await refresh_entity_summaries(session, agent_id, llm_provider=llm, config=config)
            await consolidate_entity_profiles(session, agent_id, llm_provider=llm, config=config)
            await cluster_user_facts(session, agent_id, embeddings, llm, config)
            await run_consolidation(session, agent_id, llm_provider=llm, config=config)

scheduler.add_job(maintenance_cycle, "interval", hours=4)
scheduler.start()

Example: Simple Loop

import asyncio

async def background_loop():
    while True:
        await maintenance_cycle()
        await asyncio.sleep(4 * 3600)  # every 4 hours

Recommended Cadence

Job	Frequency	Cost
Entity importance	Every 4h	Cheap (SQL only)
Summary refresh	Every 4h	Moderate (LLM, limited to 10/run)
Profile consolidation	Every 4-8h	Moderate (LLM, limited to 20/run)
Clustering	Every 4-8h	Moderate (LLM for summaries)
Consolidation	Every 4-8h	Moderate (LLM for pattern detection)
Memify	Daily	Moderate (LLM for distillation)
Community detection	Daily	Moderate (LLM + embeddings)

Run importance scoring first - its output is used by summary refresh to prioritize entities.