📊 DEVASC Topics 1.1-1.3

Data Formats, Parsing & Test-Driven Development

1.1 Compare Data Formats (XML, JSON, and YAML)

📚 Overview: XML, JSON, and YAML are three popular data serialization formats used for data exchange between applications and systems.

Format Comparison Diagram

graph TB A[Data Serialization Formats] --> B[XML] A --> C[JSON] A --> D[YAML] B --> B1[Extensible Markup Language] B --> B2[Tag-based structure] B --> B3[Schema validation] B --> B4[Verbose syntax] C --> C1[JavaScript Object Notation] C --> C2[Key-value pairs] C --> C3[Lightweight] C --> C4[Web APIs friendly] D --> D1[YAML Ain't Markup Language] D --> D2[Indentation-based] D --> D3[Human-readable] D --> D4[Configuration files] style A fill:#667eea,stroke:#764ba2,stroke-width:3px,color:#fff style B fill:#ff6b6b,stroke:#c92a2a,color:#fff style C fill:#51cf66,stroke:#2f9e44,color:#fff style D fill:#339af0,stroke:#1864ab,color:#fff

Comparison Table

Feature XML JSON YAML
Readability Moderate Good Excellent
Verbosity High Low Very Low
Data Types Text-based String, Number, Boolean, Null, Array, Object Rich types including dates
Comments <!-- comment --> Not supported # comment
Use Cases SOAP APIs, Config files REST APIs, Web apps Configuration, Ansible, K8s
File Extension .xml .json .yaml, .yml

Example Representations

XML

<?xml version="1.0"?>
<person>
    <name>John Doe</name>
    <age>30</age>
    <city>New York</city>
    <skills>
        <skill>Python</skill>
        <skill>DevOps</skill>
    </skills>
</person>
  • ✓ Schema validation
  • ✓ Attributes support
  • ✗ Verbose
  • ✗ Large file size

JSON

{
  "name": "John Doe",
  "age": 30,
  "city": "New York",
  "skills": [
    "Python",
    "DevOps"
  ]
}
  • ✓ Lightweight
  • ✓ Easy to parse
  • ✓ Web-friendly
  • ✗ No comments

YAML

name: John Doe
age: 30
city: New York
skills:
  - Python
  - DevOps
  • ✓ Most readable
  • ✓ Comments support
  • ✓ Minimal syntax
  • ✗ Whitespace sensitive
💡 Best Practices:
  • Use JSON for REST APIs and web services
  • Use YAML for configuration files and Infrastructure as Code
  • Use XML for legacy systems and when schema validation is critical

1.2 Parsing Data Formats to Python Data Structures

🔄 Overview: Python provides built-in and third-party libraries to parse XML, JSON, and YAML into native Python data structures (dictionaries, lists, strings, etc.).

Parsing Flow Diagram

graph LR A[Raw Data String/File] --> B{Format Type?} B -->|XML| C[xml.etree.ElementTree] B -->|JSON| D[json module] B -->|YAML| E[PyYAML library] C --> F[XML Tree Object] D --> G[Python Dict/List] E --> H[Python Dict/List] F --> I[Access via methods] G --> J[Access via keys] H --> K[Access via keys] style A fill:#667eea,stroke:#764ba2,color:#fff style B fill:#ffd43b,stroke:#fab005,color:#000 style C fill:#ff6b6b,stroke:#c92a2a,color:#fff style D fill:#51cf66,stroke:#2f9e44,color:#fff style E fill:#339af0,stroke:#1864ab,color:#fff

JSON Parsing

sequenceDiagram participant App as Python Application participant JSON as json Module participant Data as Python Dict/List App->>JSON: json.loads(json_string) JSON->>Data: Parse & Convert Data-->>App: Return dict/list App->>JSON: json.load(file_object) JSON->>Data: Parse from file Data-->>App: Return dict/list Note over App,Data: Serialization (Python → JSON) App->>JSON: json.dumps(python_obj) JSON-->>App: Return JSON string App->>JSON: json.dump(python_obj, file) JSON-->>App: Write to file
# JSON Parsing in Python
import json

# Parse JSON string to Python dictionary
json_string = '{"name": "Alice", "age": 25, "skills": ["Python", "APIs"]}'
data = json.loads(json_string)
print(data['name'])  # Output: Alice
print(data['skills'][0])  # Output: Python

# Parse JSON file
with open('data.json', 'r') as file:
    data = json.load(file)

# Convert Python to JSON
python_dict = {"status": "success", "code": 200}
json_output = json.dumps(python_dict, indent=2)
print(json_output)

XML Parsing

# XML Parsing in Python
import xml.etree.ElementTree as ET

# Parse XML string
xml_string = '''
<?xml version="1.0"?>
<devices>
    <device>
        <name>Router1</name>
        <ip>192.168.1.1</ip>
    </device>
</devices>
'''

root = ET.fromstring(xml_string)

# Access elements
for device in root.findall('device'):
    name = device.find('name').text
    ip = device.find('ip').text
    print(f"{name}: {ip}")

# Parse XML file
tree = ET.parse('devices.xml')
root = tree.getroot()

# Access with XPath-like syntax
devices = root.findall('.//device')
for device in devices:
    print(device.attrib)  # Get attributes

YAML Parsing

# YAML Parsing in Python
import yaml

# Parse YAML string
yaml_string = '''
server:
  host: localhost
  port: 8080
  databases:
    - name: users_db
      type: postgresql
    - name: cache_db
      type: redis
'''

data = yaml.safe_load(yaml_string)
print(data['server']['host'])  # Output: localhost
print(data['server']['databases'][0]['name'])  # Output: users_db

# Parse YAML file
with open('config.yaml', 'r') as file:
    config = yaml.safe_load(file)

# Convert Python to YAML
python_dict = {
    'version': '1.0',
    'services': ['web', 'api', 'database']
}
yaml_output = yaml.dump(python_dict, default_flow_style=False)
print(yaml_output)

Python Data Structure Mapping

graph TB subgraph JSON/YAML A1[Object/Mapping] --> P1[dict] A2[Array/Sequence] --> P2[list] A3[String] --> P3[str] A4[Number] --> P4[int/float] A5[Boolean] --> P5[bool] A6[Null/None] --> P6[None] end subgraph XML B1[Element] --> P7[Element Object] B2[Text Content] --> P8[str] B3[Attributes] --> P9[dict] end style P1 fill:#51cf66,stroke:#2f9e44,color:#fff style P2 fill:#51cf66,stroke:#2f9e44,color:#fff style P7 fill:#ff6b6b,stroke:#c92a2a,color:#fff
⚠️ Common Parsing Pitfalls:
  • JSON: Keys must be strings, no trailing commas allowed
  • XML: Namespaces can complicate parsing, use proper namespace handling
  • YAML: Indentation errors will cause parsing failures, tabs are not allowed
  • Always use try-except blocks to handle parsing errors

1.3 Test-Driven Development (TDD)

🎯 Overview: Test-Driven Development is a software development methodology where tests are written before the actual code. It follows a "Red-Green-Refactor" cycle.

TDD Cycle Diagram

graph LR A[Write Test] -->|Test Fails| B[Write Code] B -->|Test Passes| C[Refactor] C -->|Maintain Tests| A style A fill:#ff6b6b,stroke:#c92a2a,stroke-width:3px,color:#fff style B fill:#51cf66,stroke:#2f9e44,stroke-width:3px,color:#fff style C fill:#339af0,stroke:#1864ab,stroke-width:3px,color:#fff

Detailed TDD Workflow

sequenceDiagram participant Dev as Developer participant Test as Test Suite participant Code as Application Code Note over Dev,Code: RED Phase Dev->>Test: 1. Write failing test Test->>Code: Run test Code-->>Test: ❌ Test fails (expected) Test-->>Dev: Failure feedback Note over Dev,Code: GREEN Phase Dev->>Code: 2. Write minimal code Test->>Code: Run test again Code-->>Test: ✅ Test passes Test-->>Dev: Success feedback Note over Dev,Code: REFACTOR Phase Dev->>Code: 3. Improve code quality Test->>Code: Run all tests Code-->>Test: ✅ All tests still pass Test-->>Dev: Confidence to refactor

TDD Principles

🔴 RED

Write a failing test

  • Define expected behavior
  • Test should fail initially
  • Clarifies requirements
  • Ensures test validity

🟢 GREEN

Write code to pass test

  • Write minimal code
  • Focus on functionality
  • Make test pass quickly
  • Don't optimize yet

🔵 REFACTOR

Improve code quality

  • Clean up code
  • Remove duplication
  • Optimize performance
  • Keep tests passing

TDD Example in Python

# Step 1: Write the test first (RED phase)
import unittest

class TestCalculator(unittest.TestCase):
    def test_add_two_numbers(self):
        calc = Calculator()
        result = calc.add(2, 3)
        self.assertEqual(result, 5)
    
    def test_add_negative_numbers(self):
        calc = Calculator()
        result = calc.add(-1, -1)
        self.assertEqual(result, -2)

# Running this test will FAIL because Calculator doesn't exist yet

# Step 2: Write minimal code to pass (GREEN phase)
class Calculator:
    def add(self, a, b):
        return a + b

# Now the tests PASS!

# Step 3: Refactor (REFACTOR phase)
class Calculator:
    """A simple calculator with basic operations."""
    
    def add(self, a: float, b: float) -> float:
        """Add two numbers and return the result."""
        if not isinstance(a, (int, float)) or not isinstance(b, (int, float)):
            raise TypeError("Both arguments must be numbers")
        return a + b

# Tests still PASS after refactoring!

Benefits of TDD

mindmap root((TDD Benefits)) Code Quality Better design Modular code Less complexity Fewer bugs Documentation Tests as specs Usage examples Living documentation Confidence Safe refactoring Regression prevention Code coverage Development Faster debugging Clear requirements Incremental progress

Python Testing Frameworks

Framework Description Key Features
unittest Built-in Python testing framework Standard library, xUnit-style, test discovery
pytest Popular third-party framework Simple syntax, fixtures, plugins, parametrization
doctest Tests embedded in docstrings Documentation + tests, simple examples
nose2 Extends unittest Test discovery, plugins, parallel execution

Complete TDD Example: REST API Client

# test_api_client.py - Write tests first!
import unittest
from unittest.mock import patch, Mock

class TestAPIClient(unittest.TestCase):
    def setUp(self):
        self.client = APIClient("https://api.example.com")
    
    @patch('requests.get')
    def test_get_user_returns_user_data(self, mock_get):
        # Arrange
        mock_response = Mock()
        mock_response.json.return_value = {"id": 1, "name": "Alice"}
        mock_response.status_code = 200
        mock_get.return_value = mock_response
        
        # Act
        result = self.client.get_user(1)
        
        # Assert
        self.assertEqual(result['name'], "Alice")
        mock_get.assert_called_once_with("https://api.example.com/users/1")
    
    @patch('requests.get')
    def test_get_user_handles_404(self, mock_get):
        # Arrange
        mock_response = Mock()
        mock_response.status_code = 404
        mock_get.return_value = mock_response
        
        # Act & Assert
        with self.assertRaises(UserNotFoundException):
            self.client.get_user(999)

# api_client.py - Now implement the code
import requests

class UserNotFoundException(Exception):
    pass

class APIClient:
    def __init__(self, base_url):
        self.base_url = base_url
    
    def get_user(self, user_id):
        response = requests.get(f"{self.base_url}/users/{user_id}")
        
        if response.status_code == 404:
            raise UserNotFoundException(f"User {user_id} not found")
        
        response.raise_for_status()
        return response.json()

# Run tests: python -m unittest test_api_client.py
✅ TDD Best Practices:
  • Write the smallest test possible
  • One assertion per test (when practical)
  • Test behavior, not implementation
  • Keep tests fast and independent
  • Use descriptive test names
  • Follow AAA pattern: Arrange, Act, Assert
  • Run tests frequently during development
⚠️ Common TDD Mistakes:
  • Writing too many tests before implementing code
  • Testing implementation details instead of behavior
  • Skipping the refactor phase
  • Writing tests that depend on each other
  • Not running tests frequently enough

🐍 Interactive Python Interpreter

💻 Live Python Environment: Write and execute Python code directly in your browser! Try the examples below or write your own code to practice Python basics, data structures, functions, and classes.

Sample Code Templates

✏️ Python Code Editor
📤 Output
⏳ Loading Python interpreter...
✨ Features:
  • Real-time Python execution in your browser
  • Support for most standard Python 3 features
  • Practice data structures (dictionaries, lists, tuples)
  • Great for learning Python basics and DEVASC concepts
  • No installation required - runs entirely in browser
⚠️ Limitations:
  • Cannot import external libraries (json, requests, yaml, xml, etc.)
  • No file system access (no open(), read(), write())
  • Limited to basic Python features and some standard modules
  • Some advanced Python features may not work

Quick Reference

Supported Modules

  • ✓ math
  • ✓ random
  • ✓ time
  • ✓ re (regex - basic)

Supported Features

  • ✓ Lists, Dicts, Sets
  • ✓ Functions & Classes
  • ✓ Loops & Conditionals
  • ✓ List Comprehensions
  • ✓ Exception Handling

Keyboard Shortcuts

  • Tab - Indent
  • Shift+Tab - Unindent
  • Ctrl+Enter - Run Code
  • Ctrl+/ - Comment Line
← Back to Topics
← Back to Index

DEVASC 200-901 • Section 1.0: Software Development and Design (15%)

Created by Ratnesh K - CCIE x3