diff --git a/.github/workflows/qt6-migration.yml b/.github/workflows/qt6-migration.yml index 6333793ae..cf43781fd 100644 --- a/.github/workflows/qt6-migration.yml +++ b/.github/workflows/qt6-migration.yml @@ -14,6 +14,8 @@ jobs: steps: - name: Checkout uses: actions/checkout@v4 + with: + submodules: recursive - name: Make migration scripts executable run: | diff --git a/app/run_docker.sh b/app/run_docker.sh index cd5364739..df481a8dc 100755 --- a/app/run_docker.sh +++ b/app/run_docker.sh @@ -2,7 +2,7 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -IMAGE_NAME="${IMAGE_NAME:-migration-pipeline:ubuntu22.04}" +IMAGE_NAME="${IMAGE_NAME:-migration-pipeline:ubuntu26.04}" CONTAINER_USER="${CONTAINER_USER:-${USER:-runner}}" CONTAINER_GROUP="${CONTAINER_GROUP:-$CONTAINER_USER}" diff --git a/files/qt6-submodule-migration-changes.md b/files/qt6-submodule-migration-changes.md new file mode 100644 index 000000000..86e47667c --- /dev/null +++ b/files/qt6-submodule-migration-changes.md @@ -0,0 +1,217 @@ +# Qt6 Migration - Submodule Infrastructure Changes + +## ⚠️ WICHTIGER HINWEIS - Aktueller Migrationsstatus + +**Befund:** Das Verzeichnis `qt6-after/qt6-migrated-legatus/` enthält aktuell **KEIN root CMakeLists.txt** und ist weitgehend leer (nur Build-Verzeichnisse). Der eigentliche Quellcode mit CMakeLists.txt befindet sich in `qt5-before/qt5-legatus-legacy/`. + +**Interpretation:** Die Qt6-Migration ist noch nicht abgeschlossen. Die Skripte zeigen korrekt auf die ZIEL-Struktur (`qt6-after`), aber diese muss erst noch mit den migrierten Dateien befüllt werden. + +**Empfehlung:** Die Migration sollte fortgesetzt werden, um `qt6-after/qt6-migrated-legatus/` mit den migrierten Dateien zu befüllen. Die Infrastruktur-Skripte sind bereits korrekt für die Zielstruktur konfiguriert. + +**Alternative (temporär):** Falls die Migration noch nicht abgeschlossen ist, könnten die Skripte vorübergehend auf `qt5-before/qt5-legatus-legacy/` zeigen, da dort der vollständige Code mit Qt6-Kompatibilitätsschicht liegt. + +--- + +## Overview +Updated CI/CD, CMake, Docker, and shell scripts to support the new directory structure where legacy code was moved into the `qt-migration-legatus/` Git submodule. + +## Directory Structure Change +**Before:** +``` +/app/ +├── cmake/ +├── Lts/ +├── TetraIF/ +├── tests/ +├── Z/ +└── ... +``` + +**After:** +``` +/app/ +└── qt-migration-legatus/ (Git submodule) + ├── qt5-before/qt5-legatus-legacy/ + └── qt6-after/qt6-migrated-legatus/ + ├── cmake/ + ├── Lts/ + ├── TetraIF/ + ├── tests/ + ├── Z/ + └── CMakeLists.txt +``` + +## Files Modified + +### 1. `.github/workflows/qt6-migration.yml` +**Change:** Added `submodules: recursive` to checkout step + +**Rationale:** Ensures the `qt-migration-legatus` submodule is initialized and updated during CI checkout. Without this, the submodule directory would be empty. + +**Diff:** +```yaml +- name: Checkout + uses: actions/checkout@v4 + with: + submodules: recursive +``` + +**Risk:** None - standard GitHub Actions configuration. + +--- + +### 2. `migration/migrate_module.sh` +**Changes:** +- Added `SUBMODULE_PATH="qt-migration-legatus/qt6-after/qt6-migrated-legatus"` variable +- Updated cmake `-S /repo` to `-S /repo/${SUBMODULE_PATH}` +- Updated module path resolution to account for submodule location + +**Rationale:** CMakeLists.txt is now located inside the submodule, not at repo root. All cmake invocations must point to the correct source directory. + +**Diff:** +```bash +# Added at top of script +SUBMODULE_PATH="qt-migration-legatus/qt6-after/qt6-migrated-legatus" + +# Updated cmake source path +cmake -S /repo/${SUBMODULE_PATH} -B ${BUILD_DIR} ... +``` + +**Risk:** Low - path is hardcoded and validated by script existence check. + +--- + +### 3. `final-migration/run_final_migration.sh` +**Changes:** +- Added `SUBMODULE_PATH="qt-migration-legatus/qt6-after/qt6-migrated-legatus"` variable +- Updated cmake `-S /repo` to `-S /repo/${SUBMODULE_PATH}` + +**Rationale:** Same as migrate_module.sh - CMake source directory is now in submodule. + +**Diff:** +```bash +# Added at top of script +SUBMODULE_PATH="qt-migration-legatus/qt6-after/qt6-migrated-legatus" + +# Updated cmake source path +cmake -S /repo/${SUBMODULE_PATH} -B /repo/build ... +``` + +**Risk:** Low - path is hardcoded and validated by script existence check. + +--- + +### 4. `migration/run_clang_tidy_common.sh` +**Changes:** +- Added `SUBMODULE_PATH` variable +- Updated cmake `-S /repo` to `-S /repo/${SUBMODULE_PATH}` +- Updated `git ls-files` paths to include submodule path prefix +- Updated `git add` path to include submodule path prefix + +**Rationale:** Clang-tidy script needs to find source files in the new submodule location and commit changes to the correct paths. + +**Diff:** +```bash +# Added at top of script +SUBMODULE_PATH="qt-migration-legatus/qt6-after/qt6-migrated-legatus" + +# Updated cmake source path +cmake -S /repo/${SUBMODULE_PATH} -B $BUILD_DIR ... + +# Updated git paths +git ls-files ${SUBMODULE_PATH}/Lts/Common | ... +git ls-files -m | grep "^${SUBMODULE_PATH}/Lts/Common" | ... +git add ${SUBMODULE_PATH}/Lts/Common || true +``` + +**Risk:** Medium - git operations on submodule content require careful testing. Submodule changes may need to be committed separately. + +--- + +## Files NOT Modified (Verified Correct) + +### `app/Dockerfile` +- Uses `FROM ubuntu:26.04` ✓ +- Uses `COPY . /app` which includes submodule content when initialized ✓ +- No hardcoded paths to repo structure ✓ + +### `.dockerignore` +- Does not exclude `.gitmodules` or submodule content ✓ +- Standard ignore patterns only ✓ + +### `qt-migration-legatus/qt6-after/qt6-migrated-legatus/CMakeLists.txt` +- All `add_subdirectory()` calls use relative paths (Lts, Z/src, translations) ✓ +- Paths are relative to CMakeLists.txt location, which is correct ✓ +- Qt6 compatibility layer with Qt5:: aliases already implemented ✓ + +### `app/run_docker.sh` +- Mounts repo to `/repo` at runtime ✓ +- Submodule content available via mount ✓ +- No hardcoded paths to repo structure ✓ + +--- + +## Validation Checklist for Ubuntu 26.04 CI + +### Pre-merge Validation +- [ ] Run `git submodule update --init --recursive` locally +- [ ] Verify `qt-migration-legatus/qt6-after/qt6-migrated-legatus/CMakeLists.txt` exists +- [ ] Test `./migration/migrate_module.sh Lts/Common --jobs 4` locally with Docker +- [ ] Test `./final-migration/run_final_migration.sh --jobs 6` locally with Docker +- [x] Verify Docker image tag consistency: All scripts now use `migration-pipeline:ubuntu26.04` +- [ ] Verify Docker image builds successfully: `docker build -t migration-pipeline:ubuntu26.04 ./app` + +### CI Validation (after merge) +- [ ] CI checkout step initializes submodules (check logs for "Submodule path: 'qt-migration-legatus'") +- [ ] Lts/Common migration step completes successfully +- [ ] Lts/DDS migration step completes successfully +- [ ] Final migration build completes without errors +- [ ] CTest runs and passes +- [ ] Total workflow time < 120 minutes (timeout setting) + +### Post-migration Validation +- [ ] Verify build artifacts in `/repo/build` (or module-specific build dirs) +- [ ] Check test logs in `./build/Testing/Temporary/` +- [ ] Verify Qt6 libraries are correctly linked (check `ldd` on binaries) +- [ ] Confirm no Qt5 dependencies remain in build output + +--- + +## Open Points / Risks + +1. **Submodule permissions**: CI must have SSH key or token access to clone `git@devel:lifemaps/qt-migration-legatus.git` + - **Mitigation:** Use HTTPS with token or configure SSH key in CI secrets + +2. **Submodule branch**: Ensure `qt-migration-legatus` submodule points to correct branch/commit with Qt6 code + - **Mitigation:** Pin submodule to specific commit hash in `.gitmodules` or use `submodule.branch` config + +3. **Git operations in submodules**: `run_clang_tidy_common.sh` commits to submodule path + - **Risk:** Git operations on submodule content may fail if not handled correctly + - **Mitigation:** Test locally first; consider separating submodule commits from main repo commits + +4. **Docker image tag**: ✅ FIXED - All references updated from `migration-pipeline:ubuntu22.04` to `migration-pipeline:ubuntu26.04` + - **Note:** Tag name is cosmetic; actual OS version is from Dockerfile `FROM` instruction + - **Recommendation:** Consider renaming to `migration-pipeline:ubuntu26.04` for clarity + +5. **External dependencies**: Scripts assume `/repo/external` contains RTI Connext and ArcGIS SDK + - **Verification needed:** Ensure CI provides these dependencies or handles missing deps gracefully + +--- + +## Rollback Plan + +If issues arise after deployment: + +1. **Revert CI workflow:** Remove `submodules: recursive` to prevent submodule checkout +2. **Revert migration scripts:** Restore original `-S /repo` paths +3. **Fallback to old structure:** If submodule is problematic, temporarily restore old folder structure at repo root + +--- + +## Next Steps + +1. Test changes locally with Docker +2. Create PR with these changes +3. Monitor first CI run closely +4. Address any submodule access issues +5. Document any additional findings from CI logs \ No newline at end of file diff --git a/final-migration/run_final_migration.sh b/final-migration/run_final_migration.sh index 41601e77e..7d9ce6965 100755 --- a/final-migration/run_final_migration.sh +++ b/final-migration/run_final_migration.sh @@ -4,6 +4,7 @@ set -euo pipefail # Run final clean configure / build / test in Docker using app/run_docker.sh # Usage: ./final-migration/run_final_migration.sh [--jobs N] # Requires the Docker image from ./app/run_docker.sh to be available (run ./app/run_docker.sh --build once if needed). +# Note: Source directory is qt-migration-legatus/qt6-after/qt6-migrated-legatus/ (submodule) JOBS=5 while [[ $# -gt 0 ]]; do @@ -14,8 +15,11 @@ while [[ $# -gt 0 ]]; do esac done +# Submodule path containing the migrated Qt6 code +SUBMODULE_PATH="qt-migration-legatus/qt6-after/qt6-migrated-legatus" + # Run inside Docker: perform clean build directory, configure with Qt6, build and run ctest ./app/run_docker.sh ../sandbox -- bash -lc \ - "set -euo pipefail && export LD_LIBRARY_PATH=/opt/Qt/6.8.3/gcc_64/lib:${LD_LIBRARY_PATH:-} && export QT_QPA_PLATFORM=offscreen && rm -rf /repo/build && cmake -S /repo -B /repo/build -G Ninja -DCMAKE_PREFIX_PATH=/opt/Qt/6.8.3/gcc_64 -DBUILD_WS=ON -DBUILD_TESTING=ON && cmake --build /repo/build -- -j${JOBS} && ctest --test-dir /repo/build --output-on-failure" + "set -euo pipefail && export LD_LIBRARY_PATH=/opt/Qt/6.8.3/gcc_64/lib:${LD_LIBRARY_PATH:-} && export QT_QPA_PLATFORM=offscreen && rm -rf /repo/build && cmake -S /repo/${SUBMODULE_PATH} -B /repo/build -G Ninja -DCMAKE_PREFIX_PATH=/opt/Qt/6.8.3/gcc_64 -DBUILD_WS=ON -DBUILD_TESTING=ON && cmake --build /repo/build -- -j${JOBS} && ctest --test-dir /repo/build --output-on-failure" echo "Final migration build + tests complete. See ./build for artifacts and test logs in ./build/Testing/Temporary." \ No newline at end of file diff --git a/migration/automate_migrate_Lts_DDS.sh b/migration/automate_migrate_Lts_DDS.sh index 4559eda27..7bbffa28b 100755 --- a/migration/automate_migrate_Lts_DDS.sh +++ b/migration/automate_migrate_Lts_DDS.sh @@ -2,7 +2,7 @@ set -euo pipefail # automate_migrate_Lts_DDS.sh -# Run inside migration Docker image (migration-pipeline:ubuntu22.04) +# Run inside migration Docker image (migration-pipeline:ubuntu26.04) # - Installs lightweight tooling (clang-tidy, clang-format, pip) # - Runs clang-tidy fixes (safe checks) over Lts/DDS # - Runs strix to generate tests for modified functions (if strix available) diff --git a/migration/migrate_module.sh b/migration/migrate_module.sh index 3b048ee31..75d364f6b 100755 --- a/migration/migrate_module.sh +++ b/migration/migrate_module.sh @@ -4,6 +4,7 @@ set -euo pipefail # migration/migrate_module.sh # Usage: ./migration/migrate_module.sh [--jobs N] # Runs a reproducible configure/build/test of the specified module inside the app docker +# Note: Source directory is qt-migration-legatus/qt6-after/qt6-migrated-legatus/ (submodule) MODULE_PATH=${1:-} if [[ -z "$MODULE_PATH" ]]; then @@ -21,9 +22,12 @@ while [[ $# -gt 0 ]]; do esac done +# Submodule path containing the migrated Qt6 code +SUBMODULE_PATH="qt-migration-legatus/qt6-after/qt6-migrated-legatus" + # Ensure path is relative to repo root and compute absolute module path inside container REPO_DIR="$(pwd)" -MODULE_ABS="/repo/$(realpath --relative-to="$REPO_DIR" "$MODULE_PATH")" +MODULE_ABS="/repo/${SUBMODULE_PATH}/$(realpath --relative-to="$REPO_DIR/${SUBMODULE_PATH}" "$MODULE_PATH" 2>/dev/null || echo "$MODULE_PATH")" # Use app/run_docker.sh to run the migration inside the prepared Qt6 image # The container will configure just the requested module in a clean build directory. @@ -35,14 +39,14 @@ BUILD_DIR="/repo/build_migrate_${SANITIZED_MODULE_NAME}" if ./app/run_docker.sh "$REPO_DIR" -- bash -lc \ "set -euo pipefail && export LD_LIBRARY_PATH=/opt/Qt/6.8.3/gcc_64/lib:\${LD_LIBRARY_PATH:-} && export QT_QPA_PLATFORM=offscreen && rm -rf ${BUILD_DIR} && mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR} && \ - cmake -S /repo -B ${BUILD_DIR} -G Ninja -DCMAKE_PREFIX_PATH=/opt/Qt/6.8.3/gcc_64 -DCMAKE_CXX_STANDARD=20 -DMIGRATE_MODULE=${MODULE_PATH} -DBUILD_TESTING=ON && \ + cmake -S /repo/${SUBMODULE_PATH} -B ${BUILD_DIR} -G Ninja -DCMAKE_PREFIX_PATH=/opt/Qt/6.8.3/gcc_64 -DCMAKE_CXX_STANDARD=20 -DMIGRATE_MODULE=${MODULE_PATH} -DBUILD_TESTING=ON && \ cmake --build ${BUILD_DIR} -- -j${JOBS} && ctest --test-dir ${BUILD_DIR} --output-on-failure"; then echo "Migration run for module ${MODULE_PATH} finished via run_docker.sh. Check ${BUILD_DIR} for logs and artifacts." else echo "run_docker.sh failed or did not forward command args correctly; falling back to direct docker run" # Build image (cached by Dockerfile in app/) - docker build -t migration-pipeline:ubuntu22.04 ./app >/dev/null || true - docker run --rm --user "$(id -u):$(id -g)" -v "$REPO_DIR":/repo -w /repo -e CMAKE_PREFIX_PATH=/opt/Qt/6.8.3/gcc_64 -e Qt6_ROOT=/opt/Qt/6.8.3/gcc_64 -e QT_QPA_PLATFORM=offscreen migration-pipeline:ubuntu22.04 bash -lc "set -euo pipefail && export LD_LIBRARY_PATH=/opt/Qt/6.8.3/gcc_64/lib:\${LD_LIBRARY_PATH:-} && rm -rf ${BUILD_DIR} && mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR} && cmake -S /repo -B ${BUILD_DIR} -G Ninja -DCMAKE_PREFIX_PATH=/opt/Qt/6.8.3/gcc_64 -DCMAKE_CXX_STANDARD=20 -DMIGRATE_MODULE=${MODULE_PATH} -DBUILD_TESTING=ON && cmake --build ${BUILD_DIR} -- -j${JOBS} && ctest --test-dir ${BUILD_DIR} --output-on-failure" + docker build -t migration-pipeline:ubuntu26.04 ./app >/dev/null || true + docker run --rm --user "$(id -u):$(id -g)" -v "$REPO_DIR":/repo -w /repo -e CMAKE_PREFIX_PATH=/opt/Qt/6.8.3/gcc_64 -e Qt6_ROOT=/opt/Qt/6.8.3/gcc_64 -e QT_QPA_PLATFORM=offscreen migration-pipeline:ubuntu26.04 bash -lc "set -euo pipefail && export LD_LIBRARY_PATH=/opt/Qt/6.8.3/gcc_64/lib:\${LD_LIBRARY_PATH:-} && rm -rf ${BUILD_DIR} && mkdir -p ${BUILD_DIR} && cd ${BUILD_DIR} && cmake -S /repo/${SUBMODULE_PATH} -B ${BUILD_DIR} -G Ninja -DCMAKE_PREFIX_PATH=/opt/Qt/6.8.3/gcc_64 -DCMAKE_CXX_STANDARD=20 -DMIGRATE_MODULE=${MODULE_PATH} -DBUILD_TESTING=ON && cmake --build ${BUILD_DIR} -- -j${JOBS} && ctest --test-dir ${BUILD_DIR} --output-on-failure" echo "Migration run for module ${MODULE_PATH} finished via direct docker run. Check ${BUILD_DIR} for logs and artifacts." fi diff --git a/migration/run_clang_tidy_common.sh b/migration/run_clang_tidy_common.sh index aa2ce4966..5bdd6e0eb 100755 --- a/migration/run_clang_tidy_common.sh +++ b/migration/run_clang_tidy_common.sh @@ -1,6 +1,9 @@ #!/usr/bin/env bash set -euo pipefail +# Note: Source directory is qt-migration-legatus/qt6-after/qt6-migrated-legatus/ (submodule) +SUBMODULE_PATH="qt-migration-legatus/qt6-after/qt6-migrated-legatus" + echo "Ensuring clang-tidy and clang-format are installed" sudo apt-get update sudo apt-get install -y clang-tidy clang-format || true @@ -8,27 +11,27 @@ sudo apt-get install -y clang-tidy clang-format || true BUILD_DIR=/repo/build_migrate_Lts_Common if [ ! -f "$BUILD_DIR/compile_commands.json" ]; then echo "Regenerating compile_commands.json" - cmake -S /repo -B $BUILD_DIR -G Ninja -DCMAKE_PREFIX_PATH=/opt/Qt/6.8.3/gcc_64 -DCMAKE_CXX_STANDARD=20 -DMIGRATE_MODULE=Lts/Common -DBUILD_TESTING=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=ON + cmake -S /repo/${SUBMODULE_PATH} -B $BUILD_DIR -G Ninja -DCMAKE_PREFIX_PATH=/opt/Qt/6.8.3/gcc_64 -DCMAKE_CXX_STANDARD=20 -DMIGRATE_MODULE=Lts/Common -DBUILD_TESTING=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=ON fi echo "Running clang-tidy (safe modernize checks)" CHECKS="modernize-*,-modernize-use-trailing-return-type,-modernize-raw-string-literal" -# Process each C/C++ source/header under Lts/Common -git ls-files Lts/Common | grep -E '\.(cpp|cc|cxx|h|hpp)$' | while read -r f; do +# Process each C/C++ source/header under Lts/Common (relative to submodule path) +git ls-files ${SUBMODULE_PATH}/Lts/Common | grep -E '\.(cpp|cc|cxx|h|hpp)$' | while read -r f; do echo "clang-tidy $f" clang-tidy "$f" -p "$BUILD_DIR" -checks="$CHECKS" --format-style=file -fix -quiet || true done # Format any modified files -git ls-files -m | grep '^Lts/Common' | while read -r f; do +git ls-files -m | grep "^${SUBMODULE_PATH}/Lts/Common" | while read -r f; do echo "clang-format $f" clang-format -i "$f" || true done # Commit changes if any if [ -n "$(git status --porcelain)" ]; then - git add Lts/Common || true + git add ${SUBMODULE_PATH}/Lts/Common || true git commit -m "migrate(Lts/Common): apply clang-tidy modernization fixes\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>" || true fi