Planning build system simplification

This commit is contained in:
Peter Slattery 2025-07-05 13:01:30 -07:00
parent 0d6a5e4e06
commit 5f35da8cb7
2 changed files with 599 additions and 0 deletions

View File

@ -0,0 +1,203 @@
# 4coder Build System Analysis
## Overview
The 4coder build system is a sophisticated multi-tier architecture designed to handle cross-platform compilation for Windows, Linux, and macOS. The system consists of three main components:
1. **Meta-Build System**: A C++ program that generates platform-specific build commands
2. **Platform-Specific Entry Scripts**: Shell/batch scripts that compile and execute the meta-build program
3. **Custom Layer Build System**: Separate compilation process for the customizable editor layer
## Architecture Deep Dive
### 1. Meta-Build System (`code/bin/4ed_build.cpp`)
The core of the build system is a 720-line C++ program that serves as a build configuration generator. Key features:
- **Platform Detection**: Automatically detects Windows, Linux, or macOS
- **Compiler Support**: Handles CL (MSVC), GCC, and Clang compilers
- **Architecture Support**: Supports both x64 and x86 architectures
- **Build Tiers**: Manages Demo vs Super build configurations
- **Flag Management**: Centralized handling of compilation flags and options
#### Key Data Structures:
```cpp
typedef u32 Platform_Code;
enum { Platform_Windows, Platform_Linux, Platform_Mac };
typedef u32 Compiler_Code;
enum { Compiler_CL, Compiler_GCC, Compiler_Clang };
typedef u32 Arch_Code;
enum { Arch_X64, Arch_X86 };
typedef u32 Tier_Code;
enum { Tier_Demo, Tier_Super };
```
#### Build Flags System:
```cpp
enum {
OPTS = 0x1, // Compiler warnings/options
LIBS = 0x2, // Link libraries
ICON = 0x4, // Windows icon resource
SHARED_CODE = 0x8, // Build as shared library
DEBUG_INFO = 0x10, // Debug symbols
OPTIMIZATION = 0x20, // Optimization level
SUPER = 0x40, // Super tier features
INTERNAL = 0x80, // Internal build flags
SHIP = 0x100, // Shipping build
};
```
### 2. Platform-Specific Entry Scripts
Each platform has dedicated build scripts that:
1. Compile the meta-build system (`4ed_build.cpp`)
2. Execute the compiled meta-build program with appropriate flags
3. Handle platform-specific compiler setup
#### macOS (`code/bin/build-mac-dev.sh`):
```bash
#!/bin/bash
g++ -std=c++11 -DDEV_BUILD -Wno-write-strings -Wno-comment -Wno-null-dereference -Wno-logical-op-parentheses -Wno-switch -o ../build/4ed_build 4ed_build.cpp
../build/4ed_build $1
```
#### Linux (`code/bin/build-linux.sh`):
```bash
#!/bin/bash
g++ -g -std=c++11 -DDEV_BUILD -Wno-write-strings -Wno-comment -Wno-null-dereference -Wno-logical-op-parentheses -Wno-switch -o ../build/4ed_build 4ed_build.cpp
../build/4ed_build $1
```
#### Windows (`code/bin/build.bat`):
```batch
@echo off
IF NOT EXIST ..\build mkdir ..\build
pushd ..\build
cl /I..\code /DDEV_BUILD /std:c++11 /Zi /Fe:4ed_build.exe ..\code\bin\4ed_build.cpp /link /INCREMENTAL:NO /RELEASE > NUL
4ed_build.exe %1
popd
```
### 3. Custom Layer Build System
The custom layer (editor behavior/commands) uses a sophisticated multi-stage build process:
#### Stage 1: Preprocessing with Meta Macros
```bash
clang++ -I../code -I../code/custom -I../code/custom/generated -DMETA_PASS -E 4coder_default_bindings.cpp -o 4coder_default_bindings.i
```
#### Stage 2: Metadata Generator Compilation
```bash
clang++ -I../code -I../code/custom -I../code/custom/generated -o metadata_generator 4coder_metadata_generator.cpp
```
#### Stage 3: Metadata Generation
```bash
./metadata_generator -R ../code/custom/generated/ 4coder_default_bindings.i
```
#### Stage 4: Final Custom Layer Compilation
```bash
clang++ -I../code -I../code/custom -I../code/custom/generated -shared -fPIC -o custom_4coder.so 4coder_default_bindings.cpp
```
### 4. Compiler-Specific Configurations
#### Visual Studio (CL):
```cpp
#define CL_OPTS \
"-W4 -wd4310 -wd4100 -wd4201 -wd4505 -wd4996 " \
"-wd4127 -wd4510 -wd4512 -wd4610 -wd4390 " \
"-wd4611 -wd4189 -WX -GR- -EHa- -nologo -FC"
#define CL_LIBS_COMMON \
"user32.lib winmm.lib gdi32.lib opengl32.lib comdlg32.lib userenv.lib "
```
#### GCC (Linux):
```cpp
#define GCC_OPTS \
"-Wno-write-strings " \
"-D_GNU_SOURCE -fPIC " \
"-fno-threadsafe-statics -pthread " \
"-Wno-unused-result " \
"-std=c++11"
#define GCC_LIBS_COMMON \
"-lX11 -lpthread -lm -lrt " \
"-lGL -ldl -lXfixes -lfreetype -lfontconfig"
```
#### Clang (macOS):
```cpp
#define CLANG_OPTS \
"-Wno-write-strings -Wno-deprecated-declarations " \
"-Wno-comment -Wno-switch -Wno-null-dereference " \
"-Wno-tautological-compare -Wno-unused-result " \
"-Wno-missing-declarations -Wno-nullability-completeness " \
"-std=c++11 "
#define CLANG_LIBS_COMMON \
"-framework Cocoa -framework QuartzCore " \
"-framework CoreServices " \
"-framework OpenGL -framework IOKit -framework Metal -framework MetalKit "
```
### 5. Build Process Flow
#### Development Build Flow:
1. **Entry Script** (platform-specific) compiles `4ed_build.cpp`
2. **Meta-Build Program** generates compiler commands based on platform/arch/flags
3. **Custom Layer Build** (parallel process):
- Preprocesses source with meta macros
- Builds metadata generator
- Generates API bindings
- Compiles custom layer as shared library
4. **Core Engine Build**:
- Compiles `4ed_app_target.cpp` as shared library (`4ed_app.so`)
- Compiles platform layer as main executable (`4ed`)
5. **Resource Management**: Copies themes, fonts, configs to build directory
#### Packaging Build Flow:
1. **Standard Build** (as above)
2. **Distribution Preparation**:
- Creates versioned directory structure
- Copies executables and shared libraries
- Includes source code (Super tier only)
- Copies documentation and resources
3. **Archive Creation**: Creates ZIP files for distribution
### 6. Key Build Outputs
The build system produces:
- **`4ed`**: Main editor executable (platform layer)
- **`4ed_app.so`**: Core application logic (shared library)
- **`custom_4coder.so`**: Custom layer with user commands (shared library)
- **Resources**: Themes, fonts, configuration files
- **Debug Symbols**: `.dSYM` directories (macOS), `.pdb` files (Windows)
### 7. Build Complexity Analysis
The current system is complex due to:
1. **Multi-tier Architecture**: Three separate compilation units with interdependencies
2. **Cross-platform Abstraction**: Different compilers, flags, and linking requirements
3. **Dynamic API Generation**: Metadata generation for extensible command system
4. **Packaging Requirements**: Multiple build tiers and distribution formats
5. **Debug/Release Variants**: Different optimization and debugging configurations
### 8. Simplification Opportunities
The build system could be simplified by:
1. **Unified Build Scripts**: Single bash script per platform instead of meta-build system
2. **Direct Compiler Calls**: Eliminate C++ meta-build program in favor of direct clang/gcc invocations
3. **Simplified Metadata**: Reduce or eliminate runtime metadata generation
4. **Merged Compilation Units**: Combine core and custom layers into single executable
5. **Standardized Toolchain**: Use consistent compiler (clang) across all platforms
The current complexity stems from the editor's plugin architecture and cross-platform requirements, but many of these features could be maintained with a simpler build system that uses direct bash scripts calling clang with appropriate flags.

View File

@ -0,0 +1,396 @@
# 4coder Build System Simplification Plan
## Overview
This document outlines a comprehensive plan to replace the current complex meta-build system with simple, direct bash scripts that call clang with appropriate flags. The plan maintains all essential functionality while eliminating unnecessary complexity.
## Goals
1. **Simplicity**: Replace meta-build C++ program with direct bash scripts
2. **Maintainability**: Clear, readable build scripts with inline documentation
3. **Standardization**: Use clang across all platforms for consistency
4. **Functionality**: Preserve all essential build features and outputs
## Current vs Target Architecture
### Current System
```
Platform Script → Compile 4ed_build.cpp → Execute 4ed_build → Generate Commands → Build
```
### Target System
```
Platform Script → Direct Clang Calls → Build
```
## Phase 1: Foundation and Analysis (2-3 days)
### TODO 1.1: Set Up New Build Directory Structure
- [ ] Create `build_new/` directory for new build scripts
- [ ] Create subdirectories:
- `build_new/scripts/` - Main build scripts
- `build_new/helpers/` - Helper scripts and utilities
- `build_new/config/` - Build configuration files
- `build_new/temp/` - Temporary files during build
### TODO 1.2: Extract Essential Build Parameters
- [ ] Analyze `4ed_build.cpp` to extract all compiler flags per platform
- [ ] Document all linker flags and library dependencies
- [ ] Create configuration file with all essential build parameters:
```bash
# build_new/config/build-config.sh
# Platform-specific compiler flags, library paths, etc.
```
### TODO 1.3: Create Platform Detection Utility
- [ ] Write `build_new/helpers/detect-platform.sh`:
```bash
#!/bin/bash
# Returns: win32, linux, or macos
```
### TODO 1.4: Verify Clang Installation Requirements
- [ ] Document clang version requirements for each platform
- [ ] Create `build_new/helpers/check-toolchain.sh` to verify clang availability
- [ ] Document any additional platform-specific setup needed
## Phase 2: Core Build Script Development (4-5 days)
### TODO 2.1: Create Master Build Script
- [ ] Create `build_new/scripts/build.sh` as main entry point:
```bash
#!/bin/bash
# Usage: ./build.sh [platform] [config] [arch]
# Examples:
# ./build.sh macos debug x64
# ./build.sh linux release x64
# ./build.sh win32 debug x64
```
### TODO 2.2: Implement Platform-Specific Build Functions
- [ ] Create `build_new/scripts/build-macos.sh`:
```bash
#!/bin/bash
# macOS-specific build logic using clang
# Handles Objective-C++ files, frameworks, etc.
```
- [ ] Create `build_new/scripts/build-linux.sh`:
```bash
#!/bin/bash
# Linux-specific build logic using clang
# Handles X11, pthread, fontconfig, etc.
```
- [ ] Create `build_new/scripts/build-win32.sh`:
```bash
#!/bin/bash
# Windows-specific build logic using clang
# Handles Windows APIs, resource compilation, etc.
```
### TODO 2.3: Implement Core Engine Build
- [ ] Create `build_new/scripts/build-core.sh`:
```bash
#!/bin/bash
# Builds 4ed_app.so (core application logic)
# Builds 4ed executable (platform layer)
```
- [ ] Extract and implement all essential compiler flags
- [ ] Extract and implement all essential linker flags
- [ ] Handle FreeType and other external libraries
### TODO 2.4: Implement Custom Layer Build
- [ ] Create `build_new/scripts/build-custom.sh`:
```bash
#!/bin/bash
# Multi-stage custom layer build process
# 1. Preprocess with -DMETA_PASS
# 2. Build metadata generator
# 3. Generate metadata files
# 4. Compile custom layer shared library
```
- [ ] Preserve metadata generation functionality
- [ ] Preserve API binding generation
## Phase 3: Advanced Features (3-4 days)
### TODO 3.1: Implement Build Configuration System
- [ ] Create `build_new/config/flags-common.sh` - Common compiler flags
- [ ] Create `build_new/config/flags-debug.sh` - Debug-specific flags
- [ ] Create `build_new/config/flags-release.sh` - Release-specific flags
- [ ] Create `build_new/config/libs-[platform].sh` - Platform-specific libraries
### TODO 3.2: Implement Resource Management
- [ ] Create `build_new/scripts/copy-resources.sh`:
```bash
#!/bin/bash
# Copies themes, fonts, configs to build directory
# Handles platform-specific resource requirements
```
### TODO 3.3: Add Build Validation
- [ ] Create `build_new/scripts/validate-build.sh`:
```bash
#!/bin/bash
# Verifies all expected build outputs exist
# Checks library dependencies are resolved
# Validates executable functionality
```
### TODO 3.4: Implement Clean Build Support
- [ ] Create `build_new/scripts/clean.sh`:
```bash
#!/bin/bash
# Removes all build artifacts
# Supports selective cleaning (debug/release, platform-specific)
```
## Phase 4: Packaging and Distribution (2-3 days)
### TODO 4.1: Create Packaging System
- [ ] Create `build_new/scripts/package.sh`:
```bash
#!/bin/bash
# Usage: ./package.sh [platform] [tier] [arch]
# Creates distribution archives
```
- [ ] Implement directory structure creation
- [ ] Implement file copying for distribution
- [ ] Implement archive creation (ZIP format)
### TODO 4.2: Implement Version Management
- [ ] Create `build_new/scripts/version.sh`:
```bash
#!/bin/bash
# Extracts version from 4coder_version.h
# Generates version-specific naming
```
### TODO 4.3: Create Distribution Validation
- [ ] Create `build_new/scripts/validate-package.sh`:
```bash
#!/bin/bash
# Validates package contents
# Checks all required files are present
```
## Phase 5: Testing and Validation (3-4 days)
### TODO 5.1: Create Test Suite
- [ ] Create `build_new/tests/test-build.sh`:
```bash
#!/bin/bash
# Comprehensive build testing
# Tests all platform/config combinations
```
- [ ] Test debug builds on all platforms
- [ ] Test release builds on all platforms
- [ ] Test packaging on all platforms
### TODO 5.2: Performance Comparison
- [ ] Create `build_new/tests/benchmark-build.sh`:
```bash
#!/bin/bash
# Compares build times: old vs new system
# Measures clean build times
# Measures incremental build times
```
### TODO 5.3: Regression Testing
- [ ] Verify all build outputs are identical to current system
- [ ] Test custom layer loading functionality
- [ ] Test metadata generation correctness
- [ ] Test API binding generation correctness
## Phase 6: Integration and Migration (2-3 days)
### TODO 6.1: Update Project Files
- [ ] Update `code/project.4coder` to use new build scripts
- [ ] Update `code/custom/project.4coder` for custom layer builds
- [ ] Update F-key bindings to use new build system
### TODO 6.2: Create Migration Documentation
- [ ] Document differences between old and new systems
- [ ] Create migration guide for developers
- [ ] Update CLAUDE.md with new build commands
### TODO 6.3: Backup and Transition
- [ ] Create backup of current build system
- [ ] Implement gradual transition plan
- [ ] Create rollback procedure if needed
## Implementation Details
### Script Structure Template
Each build script should follow this structure:
```bash
#!/bin/bash
set -e # Exit on error
# Script: build-[platform].sh
# Purpose: Build 4coder for [platform] using clang
# Usage: ./build-[platform].sh [debug|release] [x64|x86]
# =============================================================================
# Configuration
# =============================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BUILD_ROOT="$SCRIPT_DIR/../../build"
CONFIG_DIR="$SCRIPT_DIR/../config"
# Source configuration files
source "$CONFIG_DIR/build-config.sh"
source "$CONFIG_DIR/flags-common.sh"
# =============================================================================
# Platform-specific settings
# =============================================================================
setup_platform_vars() {
# Set platform-specific variables
# - Compiler flags
# - Library paths
# - Include paths
# - Linker flags
}
# =============================================================================
# Build functions
# =============================================================================
build_core_engine() {
echo "Building core engine..."
# Document each clang call with comments
# Example:
# Build 4ed_app.so (core application logic)
clang++ \
-shared \
-std=c++11 \
-I"$CODE_DIR" \
-I"$FOREIGN_DIR/freetype2" \
-DFTECH_64_BIT \
-DFRED_SUPER \
-O2 \
-o "$BUILD_DIR/4ed_app.so" \
"$CODE_DIR/4ed_app_target.cpp" \
"$FOREIGN_DIR/x64/libfreetype-mac.a"
}
build_custom_layer() {
echo "Building custom layer..."
# Stage 1: Preprocess with meta macros
clang++ \
-I"$CODE_DIR" \
-I"$CUSTOM_DIR" \
-DMETA_PASS \
-E \
"$CUSTOM_DIR/4coder_default_bindings.cpp" \
-o "$BUILD_DIR/4coder_default_bindings.i"
# Stage 2: Build metadata generator
clang++ \
-std=c++11 \
-I"$CODE_DIR" \
-I"$CUSTOM_DIR" \
-o "$BUILD_DIR/metadata_generator" \
"$CUSTOM_DIR/4coder_metadata_generator.cpp"
# Stage 3: Generate metadata
"$BUILD_DIR/metadata_generator" \
-R "$CUSTOM_DIR/generated/" \
"$BUILD_DIR/4coder_default_bindings.i"
# Stage 4: Compile custom layer
clang++ \
-shared \
-std=c++11 \
-I"$CODE_DIR" \
-I"$CUSTOM_DIR" \
-I"$CUSTOM_DIR/generated" \
-DFTECH_64_BIT \
-DFRED_SUPER \
-O2 \
-o "$BUILD_DIR/custom_4coder.so" \
"$CUSTOM_DIR/4coder_default_bindings.cpp"
}
# =============================================================================
# Main execution
# =============================================================================
main() {
setup_platform_vars
create_build_dirs
build_core_engine
build_custom_layer
copy_resources
validate_build
echo "Build completed successfully!"
}
main "$@"
```
### Error Handling Strategy
Each script should include:
- `set -e` for exit on error
- Validation of input parameters
- Checking for required tools and dependencies
- Clear error messages with suggested fixes
- Cleanup of temporary files on failure
### Documentation Standards
Each script should include:
- Purpose and usage in header comments
- Inline documentation for each major step
- Clear variable names and consistent formatting
- Examples of usage in comments
## Success Criteria
The new build system will be considered successful when:
1. **Functionality**: All current build outputs are preserved
2. **Simplicity**: No C++ meta-build program required
3. **Maintainability**: Clear, readable bash scripts
4. **Performance**: Build times equal or better than current system
5. **Reliability**: No regression in build success rates
6. **Documentation**: Clear usage instructions and examples
## Risks and Mitigation
### Risk 1: Platform-specific Compatibility Issues
- **Mitigation**: Extensive testing on all target platforms
- **Fallback**: Maintain current system until new system is fully validated
### Risk 2: Missing Build Features
- **Mitigation**: Comprehensive analysis of current system before implementation
- **Fallback**: Gradual migration with feature parity validation
### Risk 3: Performance Regression
- **Mitigation**: Benchmark testing and optimization
- **Fallback**: Performance profiling and targeted improvements
### Risk 4: Developer Disruption
- **Mitigation**: Clear migration documentation and parallel system support
- **Fallback**: Quick rollback procedure if issues arise
## Timeline Summary
- **Phase 1**: 2-3 days - Foundation and analysis
- **Phase 2**: 4-5 days - Core build script development
- **Phase 3**: 3-4 days - Advanced features
- **Phase 4**: 2-3 days - Packaging and distribution
- **Phase 5**: 3-4 days - Testing and validation
- **Phase 6**: 2-3 days - Integration and migration
**Total Estimated Time**: 16-22 days
This plan provides a comprehensive roadmap for replacing the complex meta-build system with simple, maintainable bash scripts while preserving all essential functionality and ensuring a smooth transition for developers.