Motivation
One of the reasons that other systems are often recommended to new users is that the software required to support common devices is harder to set up and more troublesome to maintain on Fedora.
The current approach using DKMS/akmods has inherent reliability issues. With an SRE background, software should follow a build→test→deploy sequence, but installation from source (as in DKMS/akmods) follows deploy→build→test. If the build process is disrupted (power loss, reboot during silent background build, running out of disk space), or if tests fail, recovery options are limited because the software has already been deployed.
Moreover, Secure Boot presents challenges. While systems exist to support local builds with Secure Boot enabled, they require the signing key to be on the host and usable by automated processes. If your package manager can build and sign kernel modules, then a rootkit can do the same thing—defeating the purpose of Secure Boot.
Project Goals
We want:
- A system that can build and sign code for use with Secure Boot
- Using an HSM so that the signing key cannot be exfiltrated
- Providing transparency logs so that the key cannot be quietly misused
- On infrastructure that can be readily forked, deployed, discussed, and improved
The Challenge: Securing Code Signing in CI/CD
Code signing proves that binaries haven’t been tampered with and come from a trusted source. But traditional signing has a critical vulnerability: the private key must exist somewhere, and wherever it exists, it can potentially be stolen.
This project tackles that problem by building infrastructure where:
- Keys cannot be exfiltrated
- Keys cannot be used by unauthorized repositories
- Every signing operation is transparent and auditable
The Solution: Hardware-Backed Signing with Public Transparency
The architecture uses AWS KMS (Key Management Service) asymmetric keys—RSA-4096 keys backed by FIPS 140-2 Level 2 validated hardware security modules (HSMs). The private key material never leaves the HSM boundary. Ever. Not in memory, not on disk, not over the network.
Core Security Properties
1. Key Exfiltration is Cryptographically Impossible
Unlike traditional signing where a private key file exists on disk (even if encrypted), AWS KMS keys exist only within HSM boundaries:
- The private key is generated inside the HSM
- All signing operations happen inside the HSM
- The private key material is never exposed in plaintext
- Even AWS administrators cannot extract the key
- The key can be used only via authenticated AWS API calls
This means an attacker who fully compromises a build runner gets nothing. No key file to steal. No memory to dump. No credentials that grant signing access outside the controlled environment.
2. Cryptographic Transparency Logs
Every signing operation is automatically logged to a public S3 bucket via a Lambda function that processes CloudTrail events:
{
"version": "1.0",
"timestamp": "2024-01-15T12:00:00Z",
"kms_key_id": "arn:aws:kms:us-east-1:...:key/...",
"signing_algorithm": "RSASSA_PKCS1_V1_5_SHA_256",
"message_digest_sha256": "abc123...",
"signature_base64": "def456...",
"record_hash": "789ghi..."
}
These logs provide:
- Public auditability: Anyone can verify what was signed and when
- Non-repudiation: The signature proves the key owner signed the digest
- Tamper evidence: S3 versioning ensures logs cannot be silently modified
- Cryptographic proof: Each log includes the signature that can be verified with the public key
The Architecture: Defense in Depth
The system implements multiple security layers:
Network Isolation
Runners operate in a public subnet with security groups that block all inbound traffic. The runner pulls CI jobs and pushes artifacts.
VPC endpoints keep AWS service traffic within the AWS network:
- S3 endpoint: No data transfer charges, private S3 access
- KMS endpoint: KMS operations never traverse the public internet
- Secrets Manager endpoint: Runner tokens retrieved privately
- CloudWatch Logs endpoint: Monitoring traffic stays private
IAM Least Privilege
Each runner has a dedicated IAM role with minimal permissions to call kms:Sign, retrieve the public key, write to transparency logs, and read runner tokens from Secrets Manager.
Immutable Audit Trail
Multiple overlapping audit systems ensure complete accountability:
- CloudTrail: Logs every KMS API call with AWS identity, IP, timestamp
- CloudWatch Logs: Real-time streaming of signing operations
- S3 Transparency Logs: Public, versioned, immutable records
- S3 Access Logs: Track who reads the transparency logs
- Lambda Execution Logs: Record transparency log publication
Building Trust Through Transparency
This infrastructure shifts the trust model from “trust that the key is secure” to:
- Trust the cryptographic impossibility of extracting KMS keys
- Trust the mathematical proof of signatures verified by public keys
- Trust the audit trail in public transparency logs
- Trust the infrastructure-as-code that can be reviewed and reproduced
The security doesn’t depend on secrecy. The entire architecture is public—the repository, the transparency logs, the public keys. Security comes from cryptographic properties and defense in depth.
Resources
- Infrastructure Repository: https://codeberg.org/project-resistor-kernel/signed-code-build-stack
- Copr Packages: https://copr.fedorainfracloud.org/coprs/gordonmessmer/aws-kms-pkcs11/
- Kernel Repository: https://codeberg.org/project-resistor-kernel/kernel-longterm-6.12-plus