# Understanding Scoop AI

# Understanding Scoop AI

Scoop transforms natural language questions into powerful data analysis using advanced AI. This guide explains how our AI works and how to get the most from it.

## Natural Language Processing

### How Scoop Understands You

Scoop uses sophisticated language models to understand intent, not just keywords:

- **Context Awareness**: Remembers previous questions in the conversation
- **Business Terminology**: Understands industry-specific terms
- **Flexible Phrasing**: Many ways to ask the same question work equally well
- **Intent Recognition**: Knows when you want a chart vs. a number vs. an explanation

### Query Classification System

When you ask a question, Scoop instantly classifies it:

| Query Type | What It Does | Example Questions |
|------------|--------------|-------------------|
| **DATASET** | Basic data retrieval and filtering | "Show me sales last month", "List top customers" |
| **VISUALIZATION** | Creates charts and graphs | "Graph revenue over time", "Chart by region" |
| **TEXT** | General explanations and help | "What does churn mean?", "How do I export?" |
| **ML_RELATIONSHIP** | Finds factors that drive outcomes | "What predicts churn?", "What drives sales?" |
| **ML_CLUSTER** | Discovers natural groups | "Segment our customers", "Find behavior patterns" |
| **ML_GROUP** | Compares populations | "Compare gold vs silver customers" |
| **ML_PERIOD** | Analyzes changes over time | "What changed after the launch?" |
| **DEEP_REASONING** | Multi-step investigation | "Why did revenue drop?", "How can we improve?" |

## When Deep Reasoning Activates

### Automatic Triggers

Scoop automatically initiates deep reasoning for:

1. **Causal Questions**
   - "Why did..." 
   - "What caused..."
   - "What's driving..."

2. **Complex Analysis**
   - "How can we improve..."
   - "What should we do about..."
   - "What explains..."

3. **Multi-Part Questions**
   - "Why did X happen and what should we do?"
   - "What's the impact across all metrics?"

### The Analysis Choice Interface

For certain complex questions, Scoop presents you with analysis options:

```
💡 This question might benefit from deeper analysis. Choose how to proceed:

[⚡ Quick Analysis] [🧠 Deep Analysis-beta (>1min)]
```

**When You'll See This:**
- Questions with multiple valid approaches
- Queries that could benefit from thorough investigation
- Scenarios where time vs. depth is a tradeoff

**⚡ Quick Analysis**
- Results in 5-10 seconds
- Single-pass analysis
- Direct answers to your question
- Best for: Routine queries, time-sensitive decisions

**🧠 Deep Analysis-beta**
- Takes 1-3 minutes
- Multi-step investigation
- Explores multiple hypotheses
- Tests various correlations
- Provides comprehensive findings
- Best for: Root cause analysis, strategic decisions

**Example Scenario:**
```
You: Why did our conversion rate drop?

Scoop: 💡 This question might benefit from deeper analysis. Choose how to proceed:

[⚡ Quick Analysis] - Basic trend and comparison
[🧠 Deep Analysis-beta] - Full investigation with root causes
```

### What Happens During Reasoning

```
🧠 Initiating deep analysis...

Breaking down your question into investigative steps:
1. Analyze historical trends
2. Compare segments and cohorts
3. Check for correlations
4. Test statistical relationships
5. Synthesize findings

🔍 Investigating... [progress indicators]
✓ Each step completed with findings
📊 Results synthesized into coherent insights
```

## Understanding Confidence Levels

### What Confidence Means

Scoop provides confidence levels for all insights:

- **🟢 High Confidence (above 80%)**
  - Strong statistical evidence
  - Multiple supporting data points
  - Consistent patterns
  - You can act on these insights

- **🟡 Medium Confidence (50-80%)**
  - Probable relationship
  - Some supporting evidence
  - Worth investigating further
  - Consider additional validation

- **🔴 Low Confidence (below 50%)**
  - Weak correlation
  - Limited data
  - Hypothesis stage
  - Needs more investigation

### Statistical Transparency

Every ML finding includes:
- **Accuracy**: How often the model is correct
- **Sample Size**: How much data supports this
- **P-value**: Statistical significance
- **Effect Size**: Practical importance

## AI Transparency Features

### Following the Evidence Trail

Click "see why" on any finding to view:
- Supporting data points
- Statistical calculations
- Alternative explanations considered
- Confidence breakdown

### Understanding "No Pattern Found"

When Scoop reports no pattern, this means:
- ✅ **Valuable Information**: Rules out false assumptions
- ✅ **Statistical Rigor**: Prevents seeing patterns that don't exist
- ✅ **Action Insight**: Look for other factors
- ❌ **Not an Error**: This is a valid analytical result

### Model Selection Transparency

Scoop automatically selects the best approach:
- **Decision Trees (J48)**: For clear, explainable rules
- **Rule Induction (JRip)**: For if-then patterns
- **Clustering (K-means)**: For natural groupings
- **Statistical Tests**: For correlations and significance

## Getting the Most from Scoop AI

### Best Practices

1. **Start with Why**
   - ✅ "Why are sales declining?"
   - ❌ "Show sales" (too basic for AI features)

2. **Be Specific About Outcomes**
   - ✅ "What drives customer retention?"
   - ❌ "Analyze customers" (unclear goal)

3. **Include Context**
   - ✅ "Why did churn increase after our price change?"
   - ❌ "Analyze churn" (missing context)

4. **Trust the Process**
   - Let reasoning complete (10-30 seconds for complex questions)
   - Review all findings, not just the summary
   - Check confidence levels
   - Ask follow-up questions

### Progressive Analysis

Build your analysis progressively:

1. **Start Broad**: "How is the business performing?"
2. **Identify Issues**: "Revenue seems down in the West"
3. **Dig Deeper**: "Why is Western revenue declining?"
4. **Find Actions**: "What can we do to improve Western sales?"

### Combining AI Capabilities

Get comprehensive insights by combining features:

```
"Why did churn increase [reasoning] and what predicts it [ML]?"
```

This triggers both:
- Deep reasoning to understand the why
- ML analysis to predict future churn

## Common AI Patterns

### Daily Intelligence
```
"What should I know about yesterday?"
→ AI summarizes key changes, anomalies, and insights
```

### Problem Diagnosis
```
"Why are customers unhappy?"
→ Multi-step analysis across support, usage, and survey data
```

### Predictive Planning
```
"What will happen if we raise prices?"
→ ML models predict impact based on historical patterns
```

### Optimization
```
"How can we reduce costs without hurting growth?"
→ Complex reasoning balancing multiple objectives
```

## Privacy and Security

### How AI Handles Your Data

- **Processing**: All analysis happens in secure cloud environment
- **Learning**: Models don't train on your data
- **Privacy**: Each workspace is completely isolated
- **Security**: SOC2 compliant, encrypted at rest and in transit

### AI Limitations

Scoop AI is powerful but has boundaries:
- Cannot access external data not provided
- Won't make decisions requiring human judgment
- Confidence levels reflect uncertainty
- Patterns found are correlations, not always causation

## Next Steps

Now that you understand Scoop's AI:

1. **Try Deep Reasoning**: Ask a "why" question about your data
2. **Explore ML Features**: Use "what predicts" or "segment" queries  
3. **Check Confidence**: Click "see why" to understand findings
4. **Combine Approaches**: Use multiple AI features together

Remember: Scoop AI is your analytical partner, augmenting your expertise with data-driven insights you can trust.